[英]Multiprocessing pool with “apply_async” does nothing if executed from inside a function
I'm trying to use the multiprocessing
module and more partuclarly the Pool.apply_async()
function. 我正在尝试使用
multiprocessing
模块,尤其是使用Pool.apply_async()
函数。
This code works well: 该代码运行良好:
import multiprocessing
def do():
print("Foobar", flush=True)
with multiprocessing.Pool(1) as pool:
for i in range(2):
pool.apply_async(do)
pool.close()
pool.join()
The "Foobar"
string is printed twice. "Foobar"
字符串被打印两次。
However, if I put this code in a function and then call this function, nothing happens. 但是,如果我将此代码放入一个函数中,然后调用该函数,则不会发生任何事情。 No error nor
"Foobar"
, the program ends silently. 没有错误也没有
"Foobar"
,程序将以静默方式结束。
import multiprocessing
def test():
def do():
print("Foobar", flush=True)
with multiprocessing.Pool(1) as pool:
for i in range(5):
pool.apply_async(do)
pool.close()
pool.join()
test()
Why that? 为什么? I'm using Python 3.7.3 on Linux.
我在Linux上使用Python 3.7.3。
In order to retrieve your computation results do the following change to your code. 为了检索您的计算结果,请对代码进行以下更改。
import multiprocessing
def test():
def do():
print("Foobar", flush=True)
with multiprocessing.Pool(1) as pool:
for i in range(5):
result = pool.apply_async(do)
result.get()
pool.close()
pool.join()
test()
You will see the reason why "nothing happens". 您将看到“什么都没有发生”的原因。
Traceback (most recent call last):
File "/tmp/test.py", line 17, in <module>
test()
File "/tmp/test.py", line 12, in test
result.get()
File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
raise self._value
File "/usr/lib/python3.5/multiprocessing/pool.py", line 385, in _handle_tasks
put(task)
File "/usr/lib/python3.5/multiprocessing/connection.py", line 206, in send
self._send_bytes(ForkingPickler.dumps(obj))
File "/usr/lib/python3.5/multiprocessing/reduction.py", line 50, in dumps
cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'test.<locals>.do'
Python multiprocessing.Pool
relies on the pickle
protocol to serialize the data to be sent to the other process. Python
multiprocessing.Pool
依赖于pickle
协议来序列化要发送到其他进程的数据。 The pickle
protocol can serialize only top level functions and not nested ones. pickle
协议只能序列化顶级功能,而不能嵌套功能。
To see what can be pickled and what cannot check the documentation . 要查看可以腌制的内容和不能查看文档的内容 。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.