How to ensure all processors are utilized in python multiprocessing?

Question

I'm new towards multiprocessing concept.

My code

from multiprocessing import Process
def square(x):

    for x in numbers:
        print('%s squared  is  %s' % (x, x**2))

if __name__ == '__main__':
    numbers = [43, 50, 5, 98, 34, 35]

    p = Process(target=square, args=('x',))
    p.start()
    p.join
    print "Done"

result

Done
43 squared  is  1849
50 squared  is  2500
5 squared  is  25
98 squared  is  9604
34 squared  is  1156
35 squared  is  1225

I understand, we can use multiprocessing.cpu_count() to get number of cpu in system

However, I failed in achieving 2 interested things. -

1. Evenly distribute all task among all cpu
2. Check which CPU was used by which process

Answer 1

There are several things amiss in your example.

• You only start a single subprocess, which is tasked to process all the numbers.
• You are missing the parentheses from p.join() , so the process is never waited for (which is why Done is printed first).

You should instead be using multiprocessing.Pool , something like this.

from multiprocessing import Pool

def square(x):
    print('%s squared is %s' % (x, x**2))


if __name__ == '__main__':
    numbers = range(1, 1000, 50)

    with Pool() as p:
        for value in p.imap_unordered(square, numbers):
            # You could do something with the 
            # return value from `square` here.
            pass  

    print("Done")

This outputs (eg – the order is not guaranteed)

1 squared is 1
51 squared is 2601
101 squared is 10201
151 squared is 22801
201 squared is 40401
251 squared is 63001
401 squared is 160801
451 squared is 203401
501 squared is 251001
301 squared is 90601
551 squared is 303601
601 squared is 361201
351 squared is 123201
651 squared is 423801
701 squared is 491401
751 squared is 564001
801 squared is 641601
851 squared is 724201
901 squared is 811801
951 squared is 904401
Done

• Pool() defaults to using cpu_count processes, so you don't need to worry about that.
• square() only processes one number now. It should really return it for printing and processing, not print it by itself, but this is a simple example.
• You could use .map() , .imap() or some of the other methods on Pool instead; I chose .imap_unordered() because I don't care about the order in which I get those values (and besides, I'm doing nothing with them anyway).

Nothing in particular "locks" a single process to a single CPU, though – after all, a single process could be utilizing multiple threads, which the OS scheduler might schedule onto different CPUs. Different OSes have APIs to limit the processors for each process (and thread) though; if you really need that, you can dig into those.