Python Loop Behavior with Generators

Question

I am confused about why different looping constructs behave so differently when used with a simple generator. Consider the following code:

example_list = [1, 2, 3, 4, 5]
for_count = 0
next_count = 0
while_count = 0

def consumer(iterable):
    for item in iterable:
        yield    
    return

for item in consumer(example_list):
    print("For Count: {0}".format(for_count))
    for_count += 1

# First while loop
while consumer(example_list).next():
    print("Next Count: {0}".format(next_count))
    next_count += 1

# Second while loop
while consumer(example_list):
    print("While Count: {0}".format(while_count))
    while_count += 1
    if while_count > 10: # add contrived limit for infinite loop
        break

The output for this code is:

For Count: 0
For Count: 1
For Count: 2
For Count: 3
For Count: 4
While Count: 0
While Count: 1
While Count: 2
While Count: 3
While Count: 4
While Count: 5
While Count: 6
While Count: 7
While Count: 8
While Count: 9
While Count: 10

I would appreciate help understanding the following:

1. How does the for loop know when to end but not the second while loop? I expected both while loops to exit immediately since None is yielded.
2. Why doesn't using .next() raise an exception? The consumer routine isn't a class with a __next__() method defined, so does it magically appear when you use the yield keyword?
3. Why is it that if I change consumer to yield item , the first while loop become infinite like the second one?
4. If I change consumer to simply return instead of yielding anything, the second while loop exits immediate instead of becoming infinite. I've been under the impression that a yield is essentially a return that you can resume from. Why are they treated differently by a while loop?

Answer 1

The for loop

Your first for loop works as expected. Update : Mark Ransom noted that your yield is not accompanied by the expected item , so it just returns [None, None, None, None, None] rather than [1, 2, 3, 4, 5] - but it still iterates over the list.

The first while loop

The very same commentator also noticed that the first while loop never starts because 0 is a False -equivalent in Python.

The second while loop

In the second while loop, you are testing the value of consumer(example_list) . This is the generator object itself, not the values return by its next() . The object itself never equals None, or any other False equivalent - so your loop never ends.

This can be seen by printing the value of consumer(example_list) , your while condition, within the loop:

>>> while_count=0
>>> while consumer(example_list):
...     print while_count, consumer(example_list)
...     while_count += 1
...     if while_count > 10:
...         break

Giving:

0 <generator object consumer at 0x1044a1b90>
1 <generator object consumer at 0x1044a1b90>
2 <generator object consumer at 0x1044a1b90>
3 <generator object consumer at 0x1044a1b90>
4 <generator object consumer at 0x1044a1b90>
5 <generator object consumer at 0x1044a1b90>
6 <generator object consumer at 0x1044a1b90>
7 <generator object consumer at 0x1044a1b90>
8 <generator object consumer at 0x1044a1b90>
9 <generator object consumer at 0x1044a1b90>
10 <generator object consumer at 0x1044a1b90>

The second item is the object, which never equals None .

Answer 2

Answering only a subset of your question:

Your misconception with the while loops is, that a while loop does not iterate over a generator object by itself, as a for item in generator loop does.

The consumer(example_list) in your second while loop always returns a generator, which evaluates as a boolean True, hence the loops runs forever.

In the first while loop, you're testing the first yield value from that generator, which is None . Hence, the loop doesn't even start.