Processes in multiprocessing.Pool not reporting runtime errors

Question

I have a pool of processes in python using from multiprocessing import Pool . I pass to those processes different functions to be ran, calling the Pool.apply_async function. If I introduce an error in one of those functions (ie: a line of code such as 5/0 ), the expected command line report ZeroDivisionError: integer division or modulo by zero never shows up, and the program never terminates. Even if I introduce a callback function in the call to Pool.apply_async , the callback function never gets called if the function that the process has to execute has an error on it.

How can I have those processes in the pool report errors and terminate if something goes wrong?

Answer 1

You have to actually try to get the result from the AsyncResult returned by apply_async (or map_async ) for the exception to be raised in the parent.

def func():
    raise Exception("We failed")

...
result = pool.apply_async(func, args=(arg))
time.sleep(2)
result.get() # Exception only gets raised here

Any callback you provide is only executed if the function returns successfully. It gets skipped if it raises an exception.

In Python 3.2+, the error_callback keyword argument was introduced, which allows you to pass a callback that gets executed if an exception is raised in the worker, but you can't do that in Python 2.x. What you can do is use wrap your worker function in a try / except block that returns any exception raised in the worker, rather than raising it:

def func():
    try:
        raise Exception("We failed")
    except Exception as e:
        return e

Then you can have a normal callback function that checks to see if an Exception was returned:

def callback(result):
   if isinstance(result, Exception):
       # Do whatever you need to do to clean up and exit
   else:
       # Function was successful