Asynchronously schedule sequential processes with the multiprocessing.Pool Python class

Question

I'm running simulations using an executable in parallel with the Python multiprocessing.Pool class as follows:

self._pool = Pool()

return self._pool.apply_async(run_executable, [],
    dict(simulator=self,
    params=params,
    command=self._command,
    results=self._results))

Let's say we want to run 5 simulations on a computer with 4 CPUs, then (if I understand correctly) the processes get scheduled as follows:

The problem arises when I need the results of a previous simulation to start a new simulation with adjusted parameters. If we use the same figure as before and let simulation i.j+1 follow on simulation ij , the scheduling should look as follows for the easy case where we have a number of simulations equal to the number of CPUs:

How can I wait for the right process to finish, do the post-processing and then start a new simulation again in asynchronous mode? (I don't really care if this gets done by the same CPU, I just want to use the maximum amount of CPU power available).

Answer 1

Replace your code with this and have a look what it does.

self._pool = Pool()

def process_ended_callback(result):
    print(self._command, "resulted in", result)

return self._pool.apply_async(run_executable, [],
    dict(simulator=self,
    params=params,
    command=self._command,
    results=self._results), 
    process_ended_callback)

It should print which command just ended.