池回调中的异常处理

Question

Given this example scenario:

def _callback(result):
    if result == 2:
        # introduce an exception into one of the callbacks
        raise Exception("foo")

    print (result)

def _target(v):
    return v

worker_pool = Pool()

for i in range(10):
    worker_pool.apply_async(_target, args=(i,), callback=_callback)

worker_pool.close()
worker_pool.join()

I was hoping to see each value of i printed except for i=2 , which would instead have yielded an exception.

Instead I see something like the following:

0
1
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 479, in _handle_results
    cache[job]._set(i, obj)
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 649, in _set
    self._callback(self._value)
  File "test3.py", line 6, in _callback
    raise Exception("foo")
Exception: foo

... and then execution just hangs.

I'm aware that Pool handles callbacks on a separate thread, but why would execution hang and how can I reliably guard against errors in a task's callback?

Answer 1

This is happening because the exception inside the callback method is basically killing the thread that handles the Pool , as it does not have an except block to handle this kind of situation. After the Thread is dead, it's unable to join the worker_pool , so your application hangs.

I believe that's a decision made by the Python maintainers, so the best way to handle this exception is to envelop your code inside a try/except block and handle it, instead of bubbling and letting the thread be killed.