Python closures with generator

Question

def multipliers():
  return [lambda x : i * x for i in range(4)]

print [m(2) for m in multipliers()]

I partially understood(which is dangerous) the reason that i is same for all functions because Python's closures are late binding.

The output is [6, 6, 6, 6] (not [0, 2, 4, 6] as I was expecting).

---

I see that it works fine with a generator, my expected output is coming in below version.

def multipliers():
  return (lambda x : i * x for i in range(4))

print [m(2) for m in multipliers()]

Any simple explanation why it is working in the below version?

Answer 1

It only works because you call each function before the next one is created. The generator is lazy, it yields each function immediately, so before i is incremented. Compare if you force all of the generator to be consumed before you call the functions:

>>> def multipliers():
...   return (lambda x : i * x for i in range(4))
...
>>> print [m(2) for m in multipliers()]
[0, 2, 4, 6]
>>> print [m(2) for m in list(multipliers())]
[6, 6, 6, 6]

If you want early binding then you can simulate it here with default arguments:

>>> def multipliers():
...   return (lambda x, i=i : i * x for i in range(4))
...
>>> print [m(2) for m in multipliers()]
[0, 2, 4, 6]
>>> print [m(2) for m in list(multipliers())]
[0, 2, 4, 6]

To clarify my comment about the generator being lazy: the generator (lambda x : i * x for i in range(4)) will go through values of i from 0 to 3 inclusive, but it yields the first function while i is still 0, at that point it hasn't bothered to do anything about the cases for 1 to 3 (which is why we say it is lazy).

The list comprehension [m(2) for m in multipliers()] calls the first function m immediately, so i is still 0. Then the next iteration of the loop retrieves another function m where i is now 1. Again the function is called immediately so it sees i as 1. And so on.

Answer 2

You're looking for a simple explanation for a complex phenomenon, but I'll try and keep it short.

The first function returns a list of functions, each of which is a closure over the multipliers function. The interpreter therefore stores a reference to a "cell", referencing the i local variable, allowing the value to live on after the function call in which it was created has ended, and its local namespace has been destroyed.

Unfortunately, the reference in the cell is to the value of the variable at the time the function terminated, not its value at the time it was used to create the lambda (since it was used four times in a loop the interpreter would have to create a separate cell for each use, which it doesn't).

Your second function returns a generator expression, which has its own local namespace that preserves the value of the local variables (in this case, notably, i ) while suspended during the processing of a yield ed result.

You will observe that you can recast this explicitly as a generator function, which might help to explain the operation of the second example:

def multipliers():
    for i in range(4):
        yield lambda x : i * x

This too gives the required result.

Answer 3

Some points to understand this complex example:

• closure of 3 functions created point to the same i in make_fns_by_... scope
• The generator is "lazy" as explained below in details - it actually changed the code call sequence

def make_fns_by_list():
    fns = []
    for i in list(range(3)):

        def f():
            print(i)  # ref. to "global var" `i` in closure

        print(id(f), f.__closure__, f.__closure__[0].cell_contents)
        fns.append(f)

    return fns


def make_fns_by_generator():
    for i in list(range(3)):

        def f():
            print(i)  # ref. to "global var" `i` in closure

        print(id(f), f.__closure__, f.__closure__[0].cell_contents)
        yield(f)


def call_fns():

    fns = make_fns_by_generator()  # generator is lazy, do nothing here

    # for f in fns:
    #     print(id(f), f.__closure__, f.__closure__[0].cell_contents)
    # same as below which is easier for explanation:

    fns_iter = iter(fns)
    f = next(fns_iter)  # generator is "lazy", it make `f` here
    print(id(f), f.__closure__, f.__closure__[0].cell_contents, '-->', f())  # and called at once
    f = next(fns_iter)
    print(id(f), f.__closure__, f.__closure__[0].cell_contents, '-->', f())
    f = next(fns_iter)
    print(id(f), f.__closure__, f.__closure__[0].cell_contents, '-->', f())

    print('-' * 100)

    fns = make_fns_by_list()  # list is working hard, it make `f` here

    fns_iter = iter(fns)
    f = next(fns_iter)
    print(id(f), f.__closure__, f.__closure__[0].cell_contents, '-->', f())  # and called at once
    f = next(fns_iter)
    print(id(f), f.__closure__, f.__closure__[0].cell_contents, '-->', f())
    f = next(fns_iter)
    print(id(f), f.__closure__, f.__closure__[0].cell_contents, '-->', f())


def main():
    call_fns()


if __name__ == '__main__':
    main()