多处理管理器在 pool.apply_async 的非常简单的示例中失败

Question

I'm seeing some unexpected behavior in my code related to python multiprocessing , and the Manager class in particular. I wrote out a super simple example to try and better understand what's going on:

import multiprocessing as mp
from collections import defaultdict


def process(d):
    print('doing the process')
    d['a'] = []
    d['a'].append(1)
    d['a'].append(2)


def main():
    pool = mp.Pool(mp.cpu_count())
    with mp.Manager() as manager:
        d = manager.dict({'c': 2})
        result = pool.apply_async(process, args=(d))
        print(result.get())

        pool.close()
        pool.join()

        print(d)


if __name__ == '__main__':
    main()

This fails, and the stack trace printed from result.get() is as follows:

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "<string>", line 2, in __iter__
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/managers.py", line 825, in _callmethod
    proxytype = self._manager._registry[token.typeid][-1]
AttributeError: 'NoneType' object has no attribute '_registry'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "mp_test.py", line 34, in <module>
    main()
  File "mp_test.py", line 25, in main
    print(result.get())
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AttributeError: 'NoneType' object has no attribute '_registry'

I'm still unclear on what's happening here. This seems to me to be a very, very straightforward application of the Manager class. It's nearly a copy of the actual example used in the official python documentation , with the only difference being that i'm using a pool and running the process with apply_async. I'm doing this because that's what i'm using in my actual project.

To clarify, I wouldn't get a stack trace if I didn't have the result = and print(result.get()) in there. I just see {'c': 2} printed when I run the script, which indicated to me that something was going wrong and wasn't being shown.

Answer 1

A couple things to start with: first, this isn't the code you ran. The code you posted has

  result = pool.apply_async(process2, args=(d))

but there is no process2() defined. Assuming "process` was intended, the next thing is the

args=(d)

part. That's the same as typing

args=d

but that's not what's needed . You need to pass a sequence of the intended arguments. So you need to change that part to

args=(d,) # build a 1-tuple

or

args=[d]  # build a list

Then the output changes, to

{'c': 2, 'a': []}

Why aren't 1 and 2 in the the 'a' list? Because it's only the dict itself that lives on the manager server.

d['a'].append(1)

first gets the mapping for 'a' from the server, which is an empty list. But that empty list is not shared in any way - it's local to process() . You append 1 to it, and then it's thrown away - the server knows nothing about it. Same thing for 2.

To get what you want, you need to "do something" to tell the manager server about what you changed; eg,

d['a'] = L = []
L.append(1)
L.append(2)
d['a'] = L