Python multiprocessing apply_async read/write var in main process at starting subprocess

Question

I am using Python 3.5 multiprocessing apply_async. My code is like task = pool.apply_async(myFunc, args) . I pass a info (object from Info) in the args. It has a data member called startTime. I hope when myFunc starting running, info.startTime will be written as time.time() . The problem is that the info in main process and the info in subprocess is not the same. info.startTime = time.time() in myFunc does not change the info in the main process. Is there a good way to save the startTime? Thanks.

Answer 1

The processes in a pool can not write to a common variable. Think of them as existing in parallel universes. You'll need some mechanism to share information between them. Here's a simple example using Manager to keep the time stamp from all the processes:

from multiprocessing import Pool, Manager, current_process
import time

def do_work(x, ll):
    time.sleep(.2)
    ll.append(current_process().name + ' took task '+str(x)+' at '+str(time.time()))

if __name__ == '__main__':
    with Manager() as manager:
        timestamp = manager.list()
        p = Pool(processes=4)
        for x in range(10):
            p.apply_async(do_work, (x, timestamp))
        p.close()
        p.join()
        print(timestamp)

If you change timestamp = manager.list() to simply timestamp = list() , you'll see it no longer works.

PS Queue doesn't seem as easy to handle when you're using Pool .