How do closures work in runpy?

Question

I get unexpected behaviour when I try to run methods defined in a file loaded using the runpy module. The methods do not see any variables (including imported modules) defined outside of that method. Here is how I am doing it:

#test.py
import runpy
env = runpy.run_path('test', {'y':'world'})
env['fn']()

~

#test
import re

print(re.compile(r'^hello', re.IGNORECASE).sub('', "hello world"))
x = "hello"
print(x)
print(y)

def fn():
    try:
        print(re.compile(r'^hello', re.IGNORECASE).sub('', "hello world"))
    except:
        print("No re")
    try:
        print(x)
    except:
        print("No x")
    try:
        print(y)
    except:
        print("No y")

My expected output of test.py would be:

 world
hello
world
 world
hello
world

because fn would form a closure for re, x and y.

However, instead I get:

 world
hello
world
No re
None
None

It looks like re isn't defined within fn even though it should be with normal closure behaviour. x and y are even stranger because they appear to be defined but set to None.

Why is this and how do closures work with runpy? How can I achieve normal behaviour such that fn can 'see' outside variables?

Answer 1

OK, this is a curiosity of the way Python handles modules, which I know about but don't fully understand. I've come across it while working on IPython, where it's explained in a comment .

When Python runs a module, it produces a module object, the attributes of which are the global names in the module. When the module falls out of scope and is being destroyed, these attributes are set to None . Code which was defined in the function then sees these as the globals, as you found. You can demonstrate this by adding def g(): return globals() to your file, then calling env["g"]() .

I don't know if there's a way round this with runpy . IPython uses some complicated code to reuse a module object for running other files, caching copies of its __dict__ to keep the references therein alive. Have a look at themagic_run function if you're interested.