How to pass exception from one process to another?

Question

I want to raise an exception in the upload function if the running status in stop function is 'stopped'. This doesn't seem to work. I am using Pipe to pass the exception. What is wrong?

def upload(instances, u1):
    for instance in instance:
        try:
            u1.recv()
            #do_something
        except:
            #do_something_else
            continue         

def stop(instances, s1):
    for instance in instances:
        RunningStatus = instance[4]
        if RunningStatus.lower() == 'stopped'.lower():
            s1.send(Exception) # I want to raise exception in upload function 
                               # from here

if __name__ == '__main__':
    s1, u1 = multiprocessing.Pipe()
    s = multiprocessing.Process(target = stop, args = (instances, s1,))
    u = multiprocessing.Process(target = upload, args = (instances, u1))
    s.start()
    u.start()
    u.join()

Answer 1

Here's an example of how you could send Exception objects from one process to another. I also tried sending complete exception information (as returned by sys.exc_info), but, reasonably enough, this fails. One could always format the traceback info to a string and send that instead (see the traceback module).

Tested on Ubuntu 14.04 Python 2.7, 3.4 (Ubuntu-provided), and 3.5 (Continuum).

from __future__ import print_function

import sys
import multiprocessing
import time

def upload(u1):
    i=0
    try:
        while True:
            print('>upload>',i)
            i+=1
            if u1.poll():
                # tvt = u1.recv()
                # raise tvt[0], tvt[1], tvt[2] # Python 2.7
                e = u1.recv()
                raise e
            time.sleep(0.1)
    except Exception as e:
        print('Exception caught:',e)
        print('exiting')

def stop(s1):
    try:
        while True:
            for j in range(100,110):
                time.sleep(0.1)
                if 105==j:
                    raise RuntimeError("oh dear at j={}".format(j))
    except Exception as e:
        # tvt = sys.exc_info()
        # s1.send(tvt) # doesn't work; tracebacks are not pickle'able
        s1.send(e)

if __name__ == '__main__':
    s1, u1 = multiprocessing.Pipe()
    s = multiprocessing.Process(target = stop, args = (s1,))
    u = multiprocessing.Process(target = upload, args = (u1,))
    s.start()
    u.start()
    u.join()

Output:

>upload> 0
>upload> 1
>upload> 2
>upload> 3
>upload> 4
>upload> 5
>upload> 6
Exception caught: oh dear at j=105
exiting

Answer 2

You might want to look into using an Event object instead of a Pipe and Exception combination. By using an Event primitive to share data between your processes you could have the Upload function watch the event and trigger your logic when the stop function finds an issue.

Based on your example I'm assuming that each instance object in instances is an array, so by extending that array with an additional element you can have a unique event for each instance.

def upload(instances, u1):
    for instance in instance:
        if instance[5].is_set(): # events only return TRUE if they are set
            #Exception logic would go here

    else: 
        u1.recv()
        #do_something


def stop(instances, s1):
    for instance in instances:
        RunningStatus = instance[4]
        if RunningStatus.lower() == 'stopped'.lower():
            instance[5].set() # Set the event to TRUE 

if __name__ == '__main__':
    for instance in instances:
        instance[5] = multiprocessing.Event() # create a new event semaphore 
    s = multiprocessing.Process(target = stop, args = (instances, s1,))
    u = multiprocessing.Process(target = upload, args = (instances, u1))
    s.start()
    u.start()
    u.join()

The full list of multiprocessing primitives can be found over here: https://docs.python.org/2/library/multiprocessing.html#synchronization-primitives

Here's some good examples of various uses for primitives other multiprocessing objects: https://pymotw.com/2/multiprocessing/communication.html