Python and multiprocessing… how to call a function in the main process?

Question

I would like to implement an async callback style function in python... This is what I came up with but I am not sure how to actually return to the main process and call the function.

funcs = {} 

def runCallback(uniqueId):
    '''
    I want this to be run in the main process.
    '''
    funcs[uniqueId]()


def someFunc(delay, uniqueId):
    '''
    This function runs in a seperate process and just sleeps.  
    '''
    time.sleep(delay)

    ### HERE I WANT TO CALL runCallback IN THE MAIN PROCESS ###

    # This does not work... It calls runCallback in the separate process:
    runCallback(uniqueId)


def setupCallback(func, delay):
    uniqueId = id(func)
    funcs[uniqueId] = func
    proc = multiprocessing.Process(target=func, args=(delay, uniqueId))
    proc.start()
    return unqiueId

Here is how I want it to work:

def aFunc():
    return None

setupCallback(aFunc, 10)
### some code that gets run before aFunc is called ###
### aFunc runs 10s later ###

There is a gotcha here, because I want this to be a bit more complex. Basically when the code in the main process is done running... I want to examine the funcs dict and then run any of the callbacks that have not yet run. This means that runCallback also needs to remove entries from the funcs dict... the funcs dict is not shared with the seperate processes, so I think runCallback needs to be called in the main process???

Answer 1

It is unclear why do you use multiprocessing module here.

To call a function with delay in the same process you could use threading.Timer .

threading.Timer(10, aFunc).start()

Timer has .cancel() method if you'd like to cancel the callback later:

t = threading.Timer(10, runCallback, args=[uniqueId, funcs])
t.start()
timers.append((t, uniqueId))
# do other stuff
# ...
# run callbacks right now
for t, uniqueId in timers:
    t.cancel() # after this the `runCallback()` won't be called by Timer()
               # if it's not been called already
    runCallback(uniqueId, funcs)

Where runCallback() is modified to remove functions to be called:

def runCallback(uniqueId, funcs):
    f = funcs.pop(uniqueId, None) # GIL protects this code with some caveats
    if f is not None:
       f()

Answer 2

To do exactly what you're trying to do, you're going to need to set up a signal handler in the parent process to run the callback (or just remove the callback function that the child runs if it doesn't need access to any of the parent process's memory), and have the child process send a signal , but if your logic gets any more complex, you'll probably need to use another type of inter-process communication (IPC) such as pipes or sockets .

Another possibility is using threads instead of processes, then you can just run the callback from the second thread. You'll need to add a lock to synchronize access to the funcs dict.