[英]Pool.apply_async(): nested function is not executed
I am getting familiar with Python's multiprocessing
module.我开始熟悉 Python 的multiprocessing
模块。 The following code works as expected:以下代码按预期工作:
#outputs 0 1 2 3
from multiprocessing import Pool
def run_one(x):
print x
return
pool = Pool(processes=12)
for i in range(4):
pool.apply_async(run_one, (i,))
pool.close()
pool.join()
Now, however, if I wrap a function around the above code, the print
statements are not executed (or the output is redirected at least):但是,现在,如果我在上面的代码周围包装一个函数,则不会执行print
语句(或至少重定向输出):
#outputs nothing
def run():
def run_one(x):
print x
return
pool = Pool(processes=12)
for i in range(4):
pool.apply_async(run_one, (i,))
pool.close()
pool.join()
If I move the run_one
definition outside of run
, the output is the expected one again, when I'm calling run()
:如果我移动run_one
的定义之外run
,输出的预期再一个,当我打电话run()
#outputs 0 1 2 3
def run_one(x):
print x
return
def run():
pool = Pool(processes=12)
for i in range(4):
pool.apply_async(run_one, (i,))
pool.close()
pool.join()
What am I missing here?我在这里缺少什么? Why isn't the second snippet printing anything?为什么第二个片段不打印任何东西? If I simply call the run_one(i)
function instead of using apply_async
, all the three codes output the same.如果我只是调用run_one(i)
函数而不是使用apply_async
,则所有三个代码输出相同。
Pool needs to pickle (serialize) everything it sends to its worker-processes.池需要腌制(序列化)它发送到其工作进程的所有内容。 Pickling actually only saves the name of a function and unpickling requires re-importing the function by name. Pickling 实际上只保存函数的名称,而 unpickling 需要按名称重新导入函数。 For that to work, the function needs to be defined at the top-level, nested functions won't be importable by the child and already trying to pickle them raises an exception:为此,该函数需要在顶级定义,嵌套函数将无法被子进程导入,并且已经尝试对它们进行腌制会引发异常:
from multiprocessing.connection import _ForkingPickler
def run():
def foo(x):
pass
_ForkingPickler.dumps(foo) # multiprocessing custom pickler;
# same effect with pickle.dumps(foo)
run()
# Out:
Traceback (most recent call last):
...
AttributeError: Can't pickle local object 'run.<locals>.foo'
The reason why you don't see an exception is, because Pool
already starts catching exceptions during pickling tasks in the parent and only re-raises them when you call .get()
on the AsyncResult
object you immediately get when you call pool.apply_async()
.您没有看到异常的原因是,因为Pool
在父级中的酸洗任务期间已经开始捕获异常,并且只有在您调用pool.apply_async()
时立即获得的AsyncResult
对象上调用.get()
时才会重新引发它们pool.apply_async()
.
That's why (with Python 2) you better always use it like this, even if your target-function doesn't return anything (still returns implicit None
):这就是为什么(使用 Python 2)你最好总是这样使用它,即使你的目标函数不返回任何东西(仍然返回隐式None
):
results = [pool.apply_async(foo, (i,)) for i in range(4)]
# `pool.apply_async()` immediately returns AsyncResult (ApplyResult) object
for res in results:
res.get()
Non-async Pool-methods like Pool.map()
and Pool.starmap()
use the same (asynchronous) low-level functions under the hood like their asynchronous siblings, but they additionally call .get()
for you, so you will always see an exception with these methods.像Pool.map()
和Pool.starmap()
这样的非异步池方法在Pool.map()
使用相同的(异步)低级函数,就像它们的异步兄弟一样,但它们另外为你调用.get()
,所以你会总是看到这些方法的异常。
Python 3 has an error_callback
-parameter for asynchronous Pool-methods you can use instead to handle exceptions. Python 3 有一个用于异步池方法的error_callback
参数,您可以使用它来处理异常。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.