globals are not accessible inside the Multiprocessing python

Question

My application requires to execute some arbitrary python code as an input. When I try to execute it in the different process using the Multiprocessing module there are some issues with the globals() of the code.

Consider the following code which works as expected.

src = """import os
    def x():
        print(os.getcwd())"""

exec (src)
eval('x()')

This code works as expected without any error.

When I want to execute the same thing with multiprocessing with following code it gives me the error that it can't find the os ?

from concurrent.futures import ProcessPoolExecutor
pool = ProcessPoolExecutor(max_workers=1)

src = """import os
def x():
    print(os.getcwd())"""

def my_eval(source):
    try:
        exec (source)
        eval('x()')
    except Exception as ex:
        print(ex)

pool.submit(my_eval, src)

output in this case is

<Future at 0x7fcc4c160f28 state=running>
name 'os' is not defined

The answer given by viethtran solves the issue for me but can someone explain why do we need to pass globals in case of multiprocessing and why not in the case of normal execution?

Answer 1

It works when I added globals() as an argument for the exec() function.

from concurrent.futures import ProcessPoolExecutor
pool = ProcessPoolExecutor(max_workers=1)

src = """import os
def x():
    print(os.getcwd())"""

def my_eval(source):
    try:
        exec (source, globals())
        eval('x()')
    except Exception as ex:
        print(ex)

pool.submit(my_eval, src)

Update: I figured that the reason your reports name 'os' is not defined is because the os import is somehow put out-of-scope in the function x (your code will actually work if I move the import os inside the x function) so I decided to add globals() as an argument so that exec() can add the os import into globals() and will access it when x function is called by eval() .

If you print out the globals() before and after the exec (source, globals()) , you will notice that module os and function x will be added to the globals() dictionary.

from concurrent.futures import ProcessPoolExecutor
pool = ProcessPoolExecutor(max_workers=1)

src = """import os
def x():
    print(os.getcwd())"""

def my_eval(source):
    try:
        print('==========')
        print(globals())
        print('==========')
        exec(source, globals())
        print('++++++++++')
        print(globals())
        print('++++++++++')
        eval('x()')
    except Exception as ex:
        print(ex)

pool.submit(my_eval, src)

This code will print out something like this:

==========
{'my_eval': <function my_eval at 0x7ff63c7baaa0>, 'ProcessPoolExecutor': <class 'concurrent.futures.process.ProcessPoolExecutor'>, 'src': 'import os\ndef x():\n    print(os.getcwd())', '__builtins__': <module '__builtin__' (built-in)>, '__file__': 'test.py', '__package__': None, '__name__': '__main__', '__doc__': None, 'pool': <concurrent.futures.process.ProcessPoolExecutor object at 0x7ff640c14050>}
==========
++++++++++
{'my_eval': <function my_eval at 0x7ff63c7baaa0>, 'ProcessPoolExecutor': <class 'concurrent.futures.process.ProcessPoolExecutor'>, 'src': 'import os\ndef x():\n    print(os.getcwd())', '__builtins__': <module '__builtin__' (built-in)>, '__file__': 'test.py', '__package__': None, 'x': <function x at 0x7ff63c7bac80>, '__name__': '__main__', 'os': <module 'os' from '/home/viet/anaconda2/lib/python2.7/os.pyc'>, '__doc__': None, 'pool': <concurrent.futures.process.ProcessPoolExecutor object at 0x7ff640c14050>}
++++++++++
<your pwd here>