I had a script that was multiprocessing fine until today. To reproduce the problem, I simplified the function that I parallelized with the one shown below:
from multiprocessing import Process, Queue
import random
def rand_num():
num = random.random()
print(num)
if __name__ == "__main__":
queue = Queue()
processes = [Process(target=rand_num, args=()) for x in range(4)]
for p in processes:
p.start()
for p in processes:
p.join()
that renders the exact same error message (repeated 4 times, which I omitted repeating for readability):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
prepare(preparation_data)
File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 262, in run_path
code, fname = _get_code_from_file(run_name, path_name)
File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 232, in _get_code_from_file
with io.open_code(fname) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/myUserName/<stdin>'
I don't know where to start debugging this error. I'm running python3.8 under mac os Catalina (homebrew install). Please help.
I faced the same problem when upgrading from Python 3.7 to 3.8. In particular now running 3.8.6 on OSX 10.15.6, Python installed by pyenv.
The advice from Darkonaut helped to resolve the issue but it's not so visible so let me rephrase it here:
Python 3.8 on MacOS by default now uses spawn
instead of fork
as start method for new processes. Try with
multiprocessing.set_start_method("fork")
Apparently the behaviour of the spawn is wrong as following simple example reveals:
import multiprocessing
def parallel_function(x):
print("Function called with", x)
def test_pool():
print("Running test_pool")
with multiprocessing.Pool(4) as pool:
pool.map(parallel_function, range(10))
print("Starting the test")
test_pool()
This produces following output:
Starting the test
Running test_pool
Starting the test
Running test_pool
Starting the test
Running test_pool
Starting the test
Running test_pool
Starting the test
Running test_pool
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/karel/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/Users/karel/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
prepare(preparation_data)
File "/Users/karel/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/Users/karel/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
So the Pool doesn't create workers properly but instead attempts to run the whole script in each spawned process.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.