Restrict the number of processors used in multiprocessing

Question

For my code, I need to use multiprocessing module in Python to implement parallelism in the code. I have written the following code for that:

for j in range(0, len(filters)):
        p = multiprocessing.Process(target=task, args=(filters[j],j+1,img,i+1,fname))
        p.start()
        processes.append(p)

for j in range(0, len(filters)):
        p.join()

The above code works fine but it uses all the available processors in the system.

For ex: If I have 16 processors, it uses all the 16 processors in the system.

Is there any way by which I can control/limit the number of processors used by the MultiProcessing module ?

Answer 1

You should use multiprocessing.Pool - it gives you a pool of a certain size.

processes = []
with Pool(processes=4) as pool:
    for j in range(0, len(filters)):
        p = pool.apply_async(target=task, args=(filters[j],j+1,img,i+1,fname))
        processes.add(p)

    for result in processes:
        print('\t', result.get())

The full documentation is here .

This has the added benefit that you are not starting a new process for each task, but reuse the same ones. Given that starting a process is expensive, you will get better performance.

The number of processes to guess is not trivial - it depends wether your work is CPU bound, I/O bound and what load is on your PC from other programs. If you are CPU bound, you can get the number of cores like this:

multiprocessing.cpu_count()

You should probably choose a value less than that, eg -2 to leave space for other work, but that's- just a guess.