python multiprocessing pool: how can I know when all the workers in the pool have finished?

Question

I am running a multiprocessing pool in python, where I have ~2000 tasks, being mapped to 24 workers with the pool. each task creates a file based on some data analysis and webservices.

I want to run a new task, when all the tasks in the pool were finished. how can I tell when all the processes in the pool have finished?

Answer 1

You want to use the join method , which halts the main process thread from moving forward until all sub-processes ends:

Block the calling thread until the process whose join() method is called terminates or until the optional timeout occurs.

from multiprocessing import Process

def f(name):
    print 'hello', name

if __name__ == '__main__':
    processes = []
    for i in range(10):
        p = Process(target=f, args=('bob',))
        processes.append(p)

    for p in processes:
        p.start()
        p.join()

     # only get here once all processes have finished.
     print('finished!')

EDIT:

To use join with pools

    pool = Pool(processes=4)  # start 4 worker processes
    result = pool.apply_async(f, (10,))  # do some work
    pool.close()
    pool.join()  # block at this line until all processes are done
    print("completed")

Answer 2

You can use the wait() method of the AsyncResult object (which is what apply_async returns).

import multiprocessing

def create_file(i):
    open(f'{i}.txt', 'a').close()

if __name__ == '__main__':
    # The default for n_processes is the detected number of CPUs
    with multiprocessing.Pool() as pool:

        # Launch the first round of tasks, building a list of AsyncResult objects
        results = [pool.apply_async(create_file, (i,)) for i in range(50)]
    
        # Wait for every task to finish
        [result.wait() for result in results]

        # {start your next task... the pool is still available}

    # {when you reach here, the pool is closed}

This method works even if you're planning on using your pool again and don't want to close it, as @dano pointed out might be the case. For example, you might need to keep it around for the next iteration of an algorithm. Just be sure to close it when you're done.