[英]error with module multiprocessing under python3.8
我有一个直到今天都可以进行多处理的脚本。 为了重现这个问题,我简化了与下面显示的函数并行化的函数:
from multiprocessing import Process, Queue
import random
def rand_num():
num = random.random()
print(num)
if __name__ == "__main__":
queue = Queue()
processes = [Process(target=rand_num, args=()) for x in range(4)]
for p in processes:
p.start()
for p in processes:
p.join()
呈现完全相同的错误消息(重复 4 次,为了可读性我省略了重复):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
prepare(preparation_data)
File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 262, in run_path
code, fname = _get_code_from_file(run_name, path_name)
File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 232, in _get_code_from_file
with io.open_code(fname) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/myUserName/<stdin>'
我不知道从哪里开始调试这个错误。 我在 mac os Catalina(自制安装)下运行 python3.8。 请帮忙。
从 Python 3.7 升级到 3.8 时,我遇到了同样的问题。 特别是现在在 OSX 10.15.6 上运行 3.8.6,Python 由 pyenv 安装。
Darkonaut 的建议帮助解决了这个问题,但它并不那么明显,所以让我在这里重新表述:
MacOS 上的 Python 3.8 现在默认使用spawn
而不是fork
作为新进程的启动方法。 试试
multiprocessing.set_start_method("fork")
显然,spawn 的行为是错误的,如以下简单示例所示:
import multiprocessing
def parallel_function(x):
print("Function called with", x)
def test_pool():
print("Running test_pool")
with multiprocessing.Pool(4) as pool:
pool.map(parallel_function, range(10))
print("Starting the test")
test_pool()
这会产生以下输出:
Starting the test
Running test_pool
Starting the test
Running test_pool
Starting the test
Running test_pool
Starting the test
Running test_pool
Starting the test
Running test_pool
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/karel/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/Users/karel/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
prepare(preparation_data)
File "/Users/karel/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/Users/karel/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
因此,池不会正确创建工作线程,而是尝试在每个生成的进程中运行整个脚本。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.