添加到同一字典的不同生成器返回相同的值

Question

I have just encountered a weird Python (3.7.0) behavior which I don't really understand and looks like a bug for me. I want to create a dictionary with generators, but somehow all of them return the same values. Here is a code example of what I am talking about:

import itertools

d = {
"a": [-1, 2],
"b": [1, 2],
"c": [20, 20]
}
g = dict()
g2 = dict()
for letter, values in d.items():
    g[letter] = (values[0] * values[1] * x for x in itertools.count())
    g2[letter] = [values[0] * values[1] * x for x in range(3)]

for i in range(3):
    for l, v in g.items():
        print(v.__next__())
print(g2)

From my point of view, expected output would be the same for g2 elements and g generators, however I always receive values from the latest generator:

0
0
0
400
400
400
800
800
800
{'a': [0, -2, -4], 'b': [0, 2, 4], 'c': [0, 400, 800]}

To conclude, am I doing something wrong? Or is it just a standard Python behavior?

Answer 1

This error is not due to generators. It's a scoping error.

In your generator declaration, you use the name values , although the generator is not executed until after the loop is done where values is now the last item in the list. Here is an example that reproduces your error.

for i in [1]:
    g = (i for _ in range(3))

i = 'some new value'

print(next(g)) # 'some new value'

In other words values[0] and values[1] were not bound to the generator and if the value behind the name values changes then so will the generators outputs.

This means you want a closure around your generator to store the values values[0] and values[1] . You can do that by defining your generator as a function.

import itertools

# Here is a function taht will return a generator
def gen(a, b):
    for x in itertools.count():
        yield a * b * x

d = {"a": [-1, 2], "b": [1, 2], "c": [20, 20]}

g, g2 = dict(), dict()

for letter, values in d.items():
    g[letter] = gen(values[0], values[1])
    g2[letter] = [values[0] * values[1] * x for x in range(3)]

for i in range(3):
    for l, v in g.items():
        print(next(v))

print(g2)

In fact, inline generators are rarely used for that reason. Favor the def-yield way of creating generators.

Also, do no call __next__ , use the builtin next instead.

Output

0
0
0
-2
2
400
-4
4
800
{'a': [0, -2, -4], 'b': [0, 2, 4], 'c': [0, 400, 800]}

Answer 2

Check out a fixed version of your code:

import itertools

def make_generator(values):
    return (values[0] * values[1] * x for x in itertools.count())

d = {
"a": [-1, 2],
"b": [1, 2],
"c": [20, 20]
}
g = dict()
g2 = dict()
for letter, values in d.items():
    g[letter] = make_generator(values)
    g2[letter] = [values[0] * values[1] * x for x in range(3)]

for i in range(3):
    for l, v in g.items():
        print(v.__next__())
print(g2)

It prints out:

0
0
0
-2
2
400
-4
4
800
{'a': [0, -2, -4], 'b': [0, 2, 4], 'c': [0, 400, 800]}

The point is that in your code all generators work with the same values variable in the local scope, so they all end up using the values from the last key in the dict. In my version, each generator uses correct values because each generator is created in a separate scope.

This doesn't happen with the list comprehensions in g2 because they are evaluated immediately with correct values in the local scope, and the generators are evaluated later when values has already been overwritten.