The difference is probably so small that it is worth just using the method that fits your needs better. Smells of micro-optimization.
And in this case, since all you are doing is enumerating the set and counting the set (both of which you can do with an IEnumerable), why not just leave it as an IEnumerable<>?
