I am interested in how python implements closures?
For the sake of example, consider this
def closure_test():
x = 1
def closure():
nonlocal x
x = 2
print(x)
return closure
closure_test()()
Here, the function closure_test
has a local x
which the nested function closure
captures.
When I run the program, I get the following output
2
When I see the disassembly of the function closure_test
,
2 0 LOAD_CONST 1 (1)
2 STORE_DEREF 0 (x)
3 4 LOAD_CLOSURE 0 (x)
6 BUILD_TUPLE 1
8 LOAD_CONST 2 (<code object closure at 0x7f14ac3b9500, file "<string>", line 3>)
10 LOAD_CONST 3 ('closure_test.<locals>.closure')
12 MAKE_FUNCTION 8 (closure)
14 STORE_FAST 0 (closure)
7 16 LOAD_FAST 0 (closure)
18 RETURN_VALUE
Disassembly of <code object closure at 0x7f14ac3b9500, file "<string>", line 3>:
5 0 LOAD_CONST 1 (2)
2 STORE_DEREF 0 (x)
6 4 LOAD_GLOBAL 0 (print)
6 LOAD_DEREF 0 (x)
8 CALL_FUNCTION 1
10 POP_TOP
12 LOAD_CONST 0 (None)
14 RETURN_VALUE
I see the instructions STORE_DEREF
, LOAD_DEREF
, MAKE_FUNCTION
and LOAD_CLOSURE
which I won't get if I write the whole program without functions and closures.
I think those are the instructions which are needed to use closures.
But how does Python manages this? How does it captures the variable off from the local variable table of the enclosing function? And after capturing the variable where does it live? How does the function get the access of the captured variable?
I want a complete low level understanding of how it works.
Thanks in advance
Python doesn't directly use variables the same way one might expect coming from a statically-typed language like C or Java, rather it uses names and tags instances of objects with them
In your example, closure
is simply an instance of a function with that name
It's really nonlocal
here which causes LOAD_CLOSURE
and BUILD_TUPLE
to be used as described in When is the existence of nonlocal variables checked? and further in How to define free-variable in python? and refers to x
, not the inner function named literally closure
3 4 LOAD_CLOSURE 0 (x)
For your case, nonlocal
is asserting that x
exists in the outer scope excluding globals at compile time, but is practically redundant because you immediately re-use the name docs
Example of use with a global (note the SyntaxError
is at compile time, not at runtime!)
>>> x = 3
>>> def closure_test():
... def closure():
... nonlocal x
... print(x)
... return closure
...
File "<stdin>", line 3
SyntaxError: no binding for nonlocal 'x' found
>>> def closure_test():
... def closure():
... print(x)
... return closure
...
>>> closure_test()()
3
Examples of SyntaxError
s related to invalid locals use
>>> def closure_test():
... def closure():
... nonlocal x
... x = 2
... print(x)
... return closure
...
File "<stdin>", line 3
SyntaxError: no binding for nonlocal 'x' found
>>> def closure_test():
... x = 1
... def closure():
... x = 2
... nonlocal x
... print(x)
... return closure
...
File "<stdin>", line 5
SyntaxError: name 'x' is assigned to before nonlocal declaration
Original Example without nonlocal
(note absence of BUILD_TUPLE
and LOAD_CLOSURE
!)
>>> def closure_test():
... x = 1
... def closure():
... x = 2
... print(x)
... return closure
...
>>>
>>> import dis
>>> dis.dis(closure_test)
2 0 LOAD_CONST 1 (1)
2 STORE_FAST 0 (x)
3 4 LOAD_CONST 2 (<code object closure at 0x10d8132f0, file "<stdin>", line 3>)
6 LOAD_CONST 3 ('closure_test.<locals>.closure')
8 MAKE_FUNCTION 0
10 STORE_FAST 1 (closure)
6 12 LOAD_FAST 1 (closure)
14 RETURN_VALUE
Disassembly of <code object closure at 0x10d8132f0, file "<stdin>", line 3>:
4 0 LOAD_CONST 1 (2)
2 STORE_FAST 0 (x)
5 4 LOAD_GLOBAL 0 (print)
6 LOAD_FAST 0 (x)
8 CALL_FUNCTION 1
10 POP_TOP
12 LOAD_CONST 0 (None)
14 RETURN_VALUE
Reducing your example to remove all the names, it's simply
>>> import dis
>>> dis.dis(lambda: print(2))
1 0 LOAD_GLOBAL 0 (print)
2 LOAD_CONST 1 (2)
4 CALL_FUNCTION 1
6 RETURN_VALUE
The rest of the bytecode just moves the names around
x
for 1
and 2
closure
and closure_test.<locals>.closure
for inner function (located at some memory address)print
literally the print functionNone
literally the None
singleton Specific DIS opcodes
STORE_DEREF
puts a value in slot i
LOAD_DEREF
retrieves a value from slot i
MAKE_FUNCTION
creates a new function on the stack and puts it in slot i
LOAD_CLOSURE
does just that, putting it on the stack at i
You can see the constants, names, and free variables with dis.show_code()
>>> dis.show_code(closure_test)
Name: closure_test
Filename: <stdin>
Argument count: 0
Positional-only arguments: 0
Kw-only arguments: 0
Number of locals: 1
Stack size: 3
Flags: OPTIMIZED, NEWLOCALS
Constants:
0: None
1: 1
2: <code object closure at 0x10db282f0, file "<stdin>", line 3>
3: 'closure_test.<locals>.closure'
Variable names:
0: closure
Cell variables:
0: x
Digging at the closure itself
>>> dis.show_code(closure_test()) # call outer
Name: closure
Filename: <stdin>
Argument count: 0
Positional-only arguments: 0
Kw-only arguments: 0
Number of locals: 0
Stack size: 2
Flags: OPTIMIZED, NEWLOCALS, NESTED
Constants:
0: None
1: 2
Names:
0: print
Free variables:
0: x
>>> dis.show_code(lambda: print(2))
Name: <lambda>
Filename: <stdin>
Argument count: 0
Positional-only arguments: 0
Kw-only arguments: 0
Number of locals: 0
Stack size: 2
Flags: OPTIMIZED, NEWLOCALS, NOFREE
Constants:
0: None
1: 2
Names:
0: print
Tangentially-related question How does exec work with locals?
Using Python 3.9.10
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.