[英]How to solve multiprocessing stop working problem in Python 3.7.2 using venv
2019-01-12 Update 2019-01-12 更新
I reinstalled Python 3.7.1 and remade venv to get everything back to work.我重新安装了 Python 3.7.1 并重新制作了 venv 以使一切恢复正常。
But still, I am unaware of what happens in 3.7.2.但是,我仍然不知道 3.7.2 中发生了什么。
I have been using multiprocessing.map_async and .apply_async in my data processing project.我一直在我的数据处理项目中使用multiprocessing.map_async和 .apply_async 。 It worked fine in python 3.6 until 3.7.1 but when I urgrade to 3.7.2 and recreated venv, the main process just hang indefinitely and subprocesses not working at all.它在 python 3.6 中运行良好,直到 3.7.1,但是当我升级到 3.7.2 并重新创建 venv 时,主进程无限期挂起,子进程根本不工作。
I am using Windows10 and PyCharm Community.我正在使用Windows10和 PyCharm 社区。
I tried both the tool inside PyCharm and 'python -m venv' to create venv, but neither worked.我尝试了 PyCharm 中的工具和 'python -m venv' 来创建 venv,但都没有奏效。 I looked for documentation in python.org and found我在 python.org 中查找文档并找到
https://docs.python.org/3.7/whatsnew/changelog.html#python-3-7-2-final https://docs.python.org/3.7/whatsnew/changelog.html#python-3-7-2-final
It says,它说,
"venv on Windows will now use a python.exe redirector rather than copying the actual binaries from the base environment." “Windows 上的 venv 现在将使用 python.exe 重定向器,而不是从基本环境复制实际的二进制文件。”
I wonder if this has caused the problem.我想知道这是否导致了问题。
Example codes are as follows:示例代码如下:
from multiprocessing import freeze_support, Pool
def test_func(x):
y = x + 1
return y
if __name__ == '__main__':
freeze_support()
test_data = list(range(10))
with Pool(4) as test_pool:
for test_datum in test_data:
apply_result = test_pool.apply_async(test_func, test_datum)
print(apply_result.get())
I add a breakpoint in the last line and entered debug mode.我在最后一行添加了一个断点并进入调试模式。 Then I found that the apply_result object, which is a multiprocessing.pool.ApplyResult , has a _cache attribute.然后我发现apply_result对象,它是一个multiprocessing.pool.ApplyResult ,有一个_cache属性。 Under _cache there is the same multiprocessing.pool.ApplyResult but with the name of " 0 (140716767896368) ", which also has a _cache attribute, and on and on.在_cache 下有相同的multiprocessing.pool.ApplyResult但名称为“ 0 (140716767896368) ”,它也有一个_cache属性,等等。
I was desperate and tried possibly the simplest code (modified from official doc):我很绝望,尝试了可能最简单的代码(从官方文档修改):
from multiprocessing import Pool, freeze_support
def f(x):
return x*x
if __name__ == '__main__':
freeze_support()
p = Pool(5)
print(p.map(f, [1, 2, 3]))
It hangs still.它仍然挂着。
If I choose the system interpreter, not using venv, it works fine.如果我选择系统解释器,而不是使用 venv,它工作正常。
[1, 4, 9]
I would sincerely appreciate any help in solving this problem.我真诚地感谢任何帮助解决这个问题。
I had the same problem, on Mac and with VS Code....我在 Mac 和 VS Code 上遇到了同样的问题....
So here is my solution.所以这是我的解决方案。
import joblib
from joblib import Parallel,delayed
def f(x):
return x*x
number_of_cpu = joblib.cpu_count()
delayed_funcs = [delayed(f)(x) for x in [1,2,3]]
parallel_pool = Parallel(n_jobs=number_of_cpu,prefer="processes")
print(parallel_pool(delayed_funcs))
the doc is well documented anyway...无论如何,该文档都有详细记录...
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.