Why would it throws “'module' object has no attribute XXX” error when I call on apply_async from multiprocessing.Pool?

Question

The code is as below. When I copy-and-paste it in my cmd prompt, it throws 'module' object has no attribute 'func' , but when I save it as a .py file and execute python test.py , it just works fine.

import multiprocessing
import time

def func(msg):
    for i in xrange(3):
        print msg
        time.sleep(1)



if __name__ == '__main__':
    pool = multiprocessing.Pool(processes=4)
    for i in xrange(5):
        msg = "hello %d" %(i)
        pool.apply_async(func, (msg, ))
    pool.close()
    pool.join()
    print "Sub-process(es) done."

Could anyone give me an explanation on the difference between in prompt and in file when running a python code? Thanks a lot!

Answer 1

This is happening because on Windows, func needs to be pickled and sent to the child process via IPC. In order for the child to unpickle func , it needs to be able to import it from the parent's __main__ module. When this happens in a normal Python script, the child can re-import your script, and __main__ will contain all the functions declared at the top-level of your script, so it works fine. However, in the interactive interpreter, functions you've defined while in the interpreter can't simply be re-imported from a file like in a normal script, so they will not be in __main__ in the child. This is more clear if you use multiprocessing.Process directly to recreate the issue:

>>> def f():
...  print "HI"
...
>>> import multiprocessing
>>> p = multiprocessing.Process(target=f)
>>> p.start()
>>> Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\python27\lib\multiprocessing\forking.py", line 381, in main
    self = load(from_parent)
  File "C:\python27\lib\pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "C:\python27\lib\pickle.py", line 858, in load
    dispatch[key](self)
  File "C:\python27\lib\pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "C:\python27\lib\pickle.py", line 1126, in find_class
    klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'f'

This way, it's more clear that pickle can't find the module. If you add some tracing to pickle.py you can see that 'module' is referring to __main__ :

def load_global(self):
    module = self.readline()[:-1]
    name = self.readline()[:-1]
    print("module {} name {}".format(module, name))  # I added this.
    klass = self.find_class(module, name)
    self.append(klass)

Rrerunning the same code again with that extra print statement yields this:

module multiprocessing.process name Process
module __main__ name f
< same traceback as before>

It's worth noting that this example actually works fine on Posix platforms, because os.fork() is used to spawn the child processes, which means that any function defined prior to the Pool being created will be available in the child's __main__ module. So, while the above example will work, this one will still fail, because the worker function is defined after creating the Pool (which means after os.fork() is called):

>>> import multiprocessing
>>> p = multiprocessing.Pool(2)
>>> def f(a):
...  print(a)
... 
>>> p.apply(f, "hi")
Process PoolWorker-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/multiprocessing/process.py", line 231, in _bootstrap
    self.run()
  File "/usr/lib64/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python2.6/multiprocessing/pool.py", line 57, in worker
    task = get()
  File "/usr/lib64/python2.6/multiprocessing/queues.py", line 339, in get
    return recv()
AttributeError: 'module' object has no attribute 'f'