Python multiprocessing - Allocate a new function to a finished process?

Question

I have a list of 800 image files that I would like to process in parallel. Assume I store their names in a list as such:

lis_fnames = ['im1.jpg','im2.jpg',...']

Then I import the multiprocessing module and from that I import Pool. I am giving each thread a function called 'run_cli' which takes in the filename.

As of now, I run the following code:

def run_pool():
    pool = Pool(processes=4)
    pool.map(run_cli, ['im1.jpg','im2.jpg','im3.jpg','im4.jpg'])

And manually change the filenames after an iteration is complete. My question is:

Q) Given four processes, if one process has completed, how can I automatically supply it with the 'run_cli' function and another filename (from the lis_fnames list) for it to analyze, without having to wait for all four processes to be completed, and then manually run it again?

Answer 1

You should leave that to the pool. There is no need to split your iterable manually to suitable chunks. Just map it all and Pool will keep feeding your workers more work as long as there are items left in your list.

This is a simple but working example. It also shows two possible ways of handling results. It does the callback and it parses result object. You will of course do only one of these - or none if you do not care about return values from your workers.

import multiprocessing    
from time import sleep
import random

def completed(x):
    print("Done {}".format(x))

def worker(x):
    sleep(x)
    print("Worker completed {}".format(x))
    return(x)

p = multiprocessing.Pool(processes=4)

tasks = [random.randint(0,5) for _ in range(0,20)]

results = p.map_async(worker, tasks, callback=completed)

p.close()
p.join()
for r in results.get():
    print(r)

This generates a list of "tasks", in this case just a list of random numbers. Worker emulates work by sleeping the amount of seconds sent in as parameter. You can see workers complete, and finally when all workers have finished, you will get the "result".

There are always four parallel workers active, but the task list extends beyond that. Pool is able to take care of this.