Python generators timeit

Question

I am trying to understand Python generators in many tutorials guys tells that they are much faster then for example iterating through a list, so I give it a try, I wrote a simple code. I didn't expect that time difference can be that big, can someone explain me why? Or maybe I am doing something wrong here.

def f(limit):
    for i in range(limit):
        if(i / 7.0) % 1 == 0:
            yield i

def f1(limit):
    l = []
    for i in range(limit):
        if(i / 7.0) % 1 == 0:
            l.append(i)
    return l


t = timeit.Timer(stmt="f(50)", setup="from __main__ import f")
print t.timeit()

t1 = timeit.Timer(stmt="f1(50)", setup="from __main__ import f1")
print t1.timeit()

Results: t = 0.565694382945 t1 =11.9298217371

Answer 1

You are not comparing f and f1 fairly.

Your first test is only measuring how long it takes Python to construct a generator object. It never iterates over this object though, which means the code inside f is never actually executed (generators only execute their code when they are iterated over).

Your second test however measures how long it takes to call f1 . Meaning, it counts how long it takes the function to construct the list l , run the for-loop to completion, call list.append numerous times, and then return the result. Obviously, this will be much slower than just producing a generator object.

For a fair comparison, exhaust the generator f by converting it into a list:

t = timeit.Timer(stmt="list(f(50))", setup="from __main__ import f")

This will cause it to be iterated over entirely, which means the code inside f will now be executed.

Answer 2

You're timing how long it takes to create a generator object. Creating one doesn't actually execute any code, so you're essentially timing an elaborate way to do nothing.

After you fix that, you'll find that generators are usually slightly slower when run to completion. Their advantage is that they don't need to store all elements in memory at once, and can stop halfway through. For example, when you have a sequence of boolean values and want to check whether any of them are true, with lists you'd first compute all the values and create a list of them, before checking for truth, while with generators you can:

• Create the first boolean
• Check if it's true, and if so, stop creating booleans
• Else, create the second boolean
• Check if that one's true, and if so, stop creating booleans
• And so on.

Answer 3

https://wiki.python.org/moin/Generators has some good information under the section improved performance. Although creating a generator can take a bit of time, it offers a number of advantages.

• Uses less memory. By creating the values one by one, the whole list is never in memory.
• Less time to begin. Making a whole list takes time, while a generator can be used as soon as it creates the first value.
• Generators don't have a set ending point.

Here's a good tutorial on creating generators and iterators http://sahandsaba.com/python-iterators-generators.html . Check it out!