Python list comprehension vs generator

Question

I found this question Generators vs List Comprehension performance in Python and instead of cProfile I use timeit.

from timeit import timeit
import cProfile
         


        

Answer 1

The generator version won. The cProfile profiling simply introduced way more overhead for the genexp than the list comprehension, since it has a lot more points where the profiler butts in.

Answer 2

Generators load lazily; you have to make a call to get their next value every time you want it.

sum is an aggregate function, which operations on the entire iterable. You have to have all of the values available for it to do its work.

The reason that the list comprehension works faster is that there's only one explicit call to get the entire list, and one explicit operation to sum them all. However, with the generator, you have to get all of the items for it to to perform its aggregation, and since there's a million of them, that results in a million calls.

This is one of those cases in which being eager is better for performance.

Answer 3

In the simple case, it will be fastest to do this without a comprehension/generator:

sum(xrange(9999999))

Normally, if I need to do some sort of operation where I need to choose between a comprehension and generator expression, I do:

sum(a*b for a, b in zip(c, d))

Personally, I think that the generator expression (without the extra parenthesis¹) looks nicer and since readability counts -- This outweighs any micro performance differences between the two expressions.

Generators will frequently be faster for things like this because they avoid creating an intermediate list (and the memory allocation associated with it). The timing difference is probably more pronounced as the list gets bigger as the memory allocation and list resizing take more time for bigger lists. This isn't always the case however (It is well documented on StackOverflow that str.join works faster with lists than with generators in CPython because when str.join gets a generator, it constructs the list anyway...).

^{1You can omit the parenthesis any time you are passing a generator expression to a function as the only argument -- Which happens more frequently than you might expect...}