Multiprocessing Callbacks to Main Process - Python

Question

is there any way to make a callback to main process from the created process using multiprocessing? Say I have main.py which creates three processes using multiprocessing like below

from multiprocessing import Process
from child import createNewProcess

def callbackFunction(data):
   print(data)

if __name__ == "__main__"
   process = []
   for i in range(3):
        p = Process(target=createNewProcess, args=(callback,))
        process.append(p)
   [x.start() for x in process]

And child.py looks like

def createNewProcess(externalCB):
    # do something and call external callback
    data = 10 + 12
    externalCB(data)

I wanted to make a call to callbackFunction() which is available in main process context from the processes which got created. But in the above case it creates three new callbackFunction() objects in its own process and calls that which is in its own context.

How to make a call to main callbackFunction() context this using multiprocessing process?

Answer 1

snippet with example of using a queue to pass appropriate callback, and data back to a thread on the main process:

from multiprocessing import Process, Queue, current_process
from threading import Thread
from time import sleep

def foo_process(data, cb_queue, cb):
    print(f"data from process [{current_process().pid}]: {data}")
    sleep(.1) # time to flush stdout so print statement is usually in correct order
              # (may still be out of order on IPython)
    cb_queue.put((cb, data))

def foo_thread(data):
    print(f"data from cb_thread in process [{current_process().pid}]: {data*2}")

def callback_caller(cb_queue):
    for func, *args in iter(cb_queue.get, None): #pass None to exit thread
        func(*args)

if __name__ == "__main__":
    print(f"main pid: [{current_process().pid}]")
    q = Queue()

    p = Process(target=foo_process, args=(3.15, q, foo_thread))
    p.start()
    
    #It's advisable (if possible) to always create processes first then threads.
    #this primarily applies to using "fork", but it's good to know / good habit.
    t = Thread(target=callback_caller, args=(q,))
    t.start()

    p.join()

    q.put(None) #send shutdown signal to callback_caller
    t.join()