如何处理multiprocessing.Pool中的初始化错误？

Question

当初始化器抛出如下错误时，脚本不会停止。
我想在开始主进程之前中止操作（不要运行“ do_something”）。

from multiprocessing import Pool
import contextlib

def initializer():
    raise Exception("init failed")

def do_something(args):
    # main process
    pass

pool = Pool(1, initializer=initializer)
with contextlib.closing(pool):
    try:
        pool.map_async(do_something, [1]).get(100)
    except:
        pool.terminate()

控制台上永不停止的stacktrace在下面

...
Exception: init failed
Process ForkPoolWorker-18:
Traceback (most recent call last):
  File "/home/hoge/anaconda3/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/home/hoge/anaconda3/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/hoge/anaconda3/lib/python3.6/multiprocessing/pool.py", line 103, in worker
    initializer(*initargs)
  File "hoge.py", line 5, in initializer
    raise Exception("init failed")
Exception: init failed
...

我的解决方法是抑制初始化错误并通过使用如下所示的全局标志在主进程的开始处返回。
但我想学得更好。

def initializer():
    try:
        raise Exception("init failed")
    except:
        global failed
        failed = True

def do_something(args):
    global failed
    if failed:
        # skip when initializer failed
        return
    # main process

Answer 1

在使用PyCharm实现了多处理的实现之后，我确信没有更好的解决方案，因为如果存在任何辅助进程，则Pool会通过_repopulate_pool（）启动到_maintain_pool（）的线程-偶然或初始化失败。

检查一下： Lib / multiprocessing / pool.py第244行

Answer 2

我只是遇到了同样的麻烦。 我的第一个解决方案是捕获异常并在worker函数中引发它（请参见下文）。 但经过深思熟虑，这的确意味着multiprocessing.Pool initializer支持。 multiprocessing.Pool已损坏且无法使用。 因此，我现在更喜欢直接在worker中进行初始化工作。

from multiprocessing import Pool
import contextlib, sys

_already_inited = False
def initializer():
    global _already_inited
    if _already_inited:
        return
    _already_inited = True
    raise Exception("init failed")

def do_something(args):
    initializer()
    # main process

pool = Pool(1)
with contextlib.closing(pool):
    pool.map_async(do_something, [1]).get(100)

代码和stacktrace都更简单。 当然，您的所有辅助函数都需要调用initializer() 。

我最初的解决方案是将异常延迟给worker函数。

from multiprocessing import Pool
import contextlib, sys

failed = None
def initializer():
    try:
        raise Exception("init failed")
    except:
        global failed
        failed = sys.exc_info()[1]

def do_something(args):
    global failed
    if failed is not None:
        raise RuntimeError(failed) from failed
    # main process

pool = Pool(1, initializer=initializer)
with contextlib.closing(pool):
    pool.map_async(do_something, [1]).get(100)

这样，调用方仍然可以访问异常。

multiprocessing.pool.RemoteTraceback:  
""" 
Traceback (most recent call last): 
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 119, in worker 
    result = (True, func(*args, **kwds)) 
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar 
    return list(map(*args)) 
  File "/tmp/try.py", line 15, in do_something 
    raise RuntimeError(failed) 
RuntimeError: init failed 
""" 

The above exception was the direct cause of the following exception: 

Traceback (most recent call last): 
  File "/tmp/try.py", line 20, in <module> 
    pool.map_async(do_something, [1]).get(100) 
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get 
    raise self._value 
RuntimeError: init failed 
(venv) kmkaplan@dev1:~/src/options$ python3 /tmp/try.py  
multiprocessing.pool.RemoteTraceback:  
""" 
Traceback (most recent call last): 
  File "/tmp/try.py", line 7, in initializer 
    raise Exception("init failed") 
Exception: init failed 

The above exception was the direct cause of the following exception: 

Traceback (most recent call last): 
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 119, in worker 
    result = (True, func(*args, **kwds)) 
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar 
    return list(map(*args)) 
  File "/tmp/try.py", line 15, in do_something 
    raise RuntimeError(failed) from failed 
RuntimeError: init failed 
""" 

The above exception was the direct cause of the following exception: 

Traceback (most recent call last): 
  File "/tmp/try.py", line 20, in <module> 
    pool.map_async(do_something, [1]).get(100) 
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get 
    raise self._value 
RuntimeError: init failed