Possible reasons why Pool map is not using all available resources

Question

I'm running the following code

from multiprocessing import Pool


def loop_f(x, num_loops):
    for i in range(num_loops):
        f(x)
    return 

def f(x):
    result = 0 
    for i in range(x):
        result = result*i
    return result

x = 200000
num_times=200
for i in range(8):
    p = Pool(i +1)
    print(i+1)
    %time res=p.map(f, [x]*num_times)

Now when I run this code I see that the performance improvement stops after the 4th process

Timing when using  1  processes
CPU times: user 9.08 ms, sys: 13.4 ms, total: 22.5 ms
Wall time: 1.17 s
Timing when using  2  processes
CPU times: user 0 ns, sys: 12.1 ms, total: 12.1 ms
Wall time: 598 ms
Timing when using  3  processes
CPU times: user 5.51 ms, sys: 5.6 ms, total: 11.1 ms
Wall time: 467 ms
Timing when using  4  processes
CPU times: user 9.1 ms, sys: 479 µs, total: 9.58 ms
Wall time: 348 ms
Timing when using  5  processes
CPU times: user 4.15 ms, sys: 4.51 ms, total: 8.66 ms
Wall time: 352 ms
Timing when using  6  processes
CPU times: user 6.85 ms, sys: 2.74 ms, total: 9.59 ms
Wall time: 343 ms
Timing when using  7  processes
CPU times: user 2.79 ms, sys: 7.16 ms, total: 9.95 ms
Wall time: 349 ms
Timing when using  8  processes
CPU times: user 9.06 ms, sys: 427 µs, total: 9.49 ms
Wall time: 362 ms

But when I check my system, I should have access to at 8 processor cores.

import multiprocessing
import os

print(multiprocessing.cpu_count())
print(len(os.sched_getaffinity(0)))

8
8

So what's happening, or possibly happening? How can I maximize my system's performance?

Answer 1

You should only create a Pool once.

from multiprocessing import Pool

def f(x):
    j = 0
    for i in range(1000000):
        j += i

    return x*x

if __name__ == '__main__':
    with Pool(8) as p:
        print(p.map(f, range(1000)))

The above keeps my eight threads busy for a while.

Answer 2

My machine actually only has 4 cores: https://ark.intel.com/content/www/us/en/ark/products/75056/intel-xeon-processor-e3-1270-v3-8m-cache-3-50-ghz.html

import multiprocessing
import os

print(multiprocessing.cpu_count())
print(len(os.sched_getaffinity(0)))

Does not report the number of cores only the number of threads

Answer 3

multiprocessing.Pool() is used to declare the number of cores you want your process to run on. the different methods of Pool tell how you want to apply the multiprocessing on those processes.

The first problem in your code is, you are initializing the Pool each time with different number of cores. The second is once your pool of workers are done with the processes, you should join them.

I rewrite the code:

from multiprocessing import Pool
import multiprocessing as mp

def f(x):
    j = 0
    for i in range(1000000):
        j += i
    return x*x

if __name__=='__main__':
    p=Pool(mp.cpu_count()) #Declaring the Pools with the number of cpus your machine has
    res=p.map_async(f, range(1000))

    p.close() #close the pool
    p.join() #join all the workers
    print(res.get())