Python generator confusing me

Question

Why this happended? I am getting confused. In the first case, I define chunked_test_list by a list conprehension. The output is as expected. But, in the next case, I define chunked_test_list as a generator. And then, when I look through it with a for loop, just the first loop give me the expected result, and the rest is EMPTY list. I got confused.. And, in the last case, I change the variable name from test_list to test_list_1 in for loop, and then it turns out run as expected. Oh, I got more confused.

test_list = [1, 2, 3, 4, 5, 6, 7, 8]
step = 2

chunked_test_list = [test_list[x: x + step] for x in range(0, len(test_list), step)]
for test_list in chunked_test_list:
    print('test_list', test_list)

# >>>output
# test_list [1, 2]
# test_list [3, 4]
# test_list [5, 6]
# test_list [7, 8]

test_list = [1, 2, 3, 4, 5, 6, 7, 8]
step = 2

chunked_test_list = (test_list[x: x + step] for x in range(0, len(test_list), step))  # GENERATOR
for test_list in chunked_test_list:  # the variable name `test_list` is SAME to list above
    print('test_list', test_list)

# >>>output
# test_list [1, 2]
# test_list []
# test_list []
# test_list []

test_list = [1, 2, 3, 4, 5, 6, 7, 8]
step = 2

chunked_test_list = (test_list[x: x + step] for x in range(0, len(test_list), step))  # GENERATOR
for test_list_1 in chunked_test_list:  # the variable name `test_list` is NOT same to list above
    print('test_list_1', test_list_1)

# >>>output
# test_list_1 [1, 2]
# test_list_1 [3, 4]
# test_list_1 [5, 6]
# test_list_1 [7, 8]

Answer 1

The result you are seeing is formally caused by the binding of names in a generator, as laid out in PEP289 .

The list comprehension chunked_test_list = [test_list[x: x + step] for x in range(0, len(test_list), step)] refers to test_list , but creates a list of sliced lists immediately. Rebinding the name test_list as the subsequent loop variable does not affect the elements of the list chunked_test_list .

Now let's take a look at the generator

chunked_test_list = (test_list[x: x + step] for x in range(0, len(test_list), step))

The loop expression range(0, len(test_list), step) is evaluated immediately. The range is created as soon as the line runs, and you can see that it uses the expected len(test_list) by the number of iterations.

At the same time, the body of the generator loop, test_list[x: x + step] , is evaluated every time the outer for loop calls next . The loop for test_list in chunked_test_list: effectively performs the assignment test_list = gen.next() at each iteration. During the first call to next , test_list is bound to its original value. You see the first element of the generator print as expected. As soon as the call returns however, the name test_list is bound to the result of next . Since none of the return values are long enough to support the index generated by the range, you see empty lists.

When you rename the loop variable in the generator to test_list_1 , test_list does not get rebound, and the loop completes without conflict.