Python multiprocessing.Pool（）发出错误

Question

I am using python 2.7.14 and need to spawn a method across multiple processes. I tried using multiprocessing:

from multiprocessing import Pool
pool = Pool()

This give out the following error:

File "sw\lib\multiprocessing\__init__.py", line 232, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild)
File "sw\lib\multiprocessing\pool.py", line 61, in __init__
self._repopulate_pool()
File "sw\lib\multiprocessing\pool.py", line 25, in _repopulate_pool
w.start()
File "sw\lib\multiprocessing\process.py", line 130, in start
self._popen = Popen(self)
File "sw\lib\multiprocessing\forking.py", line 272, in __init__
prep_data = get_preparation_data(process_obj._name)
File "<console>", line 2, in _get_preparation_data
File "sw\lib\multiprocessing\forking.py", line 409, in get_preparation_data
not d['sys_argv'][0].lower().endswith('pythonservice.exe'):
IndexError: list index out of range

Is this a bug with python 2.7?

Is there an alternative in python for multiprocessing?

Answer 1

I used to experience library missing like this when installing python by Anaconda . The recommendation for this is to re-install Anaconda so that the missing package will be installed again.

Pool , Lock & Semaphore are multi-processors' wrapper that actually wrap processes to control CPU usage, RAM and other capacity of your computer. You can check this article on how to use it https://realpython.com/intro-to-python-threading/

Answer 2

I am able to use Pool in multiprocessing with my python 2.7.14. Can u try creating the same in a conda environment and see if it works

conda create -n test2.7 python=2.7.14
source activate test2.7

Now open python shell and see if it works

from multiprocessing import Pool
pool = Pool()

Answer 3

Q: Is this a bug with python 2.7?
A: No, this is not a bug,
most probably the state of your localhost installation is corrupted

OBSERVATIONS:

Your py2.7 System-under-Test (SuT):
FAIL / EXC'ed: SuT resorted to throw an Exception as it failed to find a pythonservice.exe file during a Pool() instantiation. The localhost system, not having such file, cannot proceed as was demonstrated above and threw an Exc:

File "sw\lib\multiprocessing\forking.py", line 409, in get_preparation_data
      not d['sys_argv'][0].lower().endswith('pythonservice.exe'):
IndexError: list index out of range

Standard py2.7 SuT :
PASS

Python 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Pool
>>> pool = Pool()
>>>

ASSESSMENT:

Python 2.7 interpreter is capable to execute the MCVE-code without any warning and/or errors. The problem is specific to your-py2.7-SuT, not reproducible in general and observations indicate your-py2.7-SuT is flawed, most probably due to corrupted state of the python ecosystem installation ( EXC was reporting missing some files, that were otherwise fair to be assumed to be always present is such ecosystem ).

QED

Q: Is there an alternative in python for multiprocessing?

The proper step is to repair ( by a full reinstall ) the localhost python installation, as it has shown itself to be corrupted.

Using any of the contemporary package management framework ( mini-conda, conda et al ) is strongly encouraged, so as to be sure of proper configuration management and version control tools are used to prevent your new installation from mixed-version incompatibilities and any similar unwanted side-effects from re-appearance.