multiprocessing full capacity in Python

Question

I wrote the following code which call function (compute_cluster) 6 times in parallel (each run of this function is independent of the other run and each run write the results in a separate file), the following is my code:

global L
for L in range(6,24):
       pool = Pool(6)
       pool.map(compute_cluster,range(1,3)) 
       pool.close()  

if __name__ == "__main__":
   main(sys.argv)            

despite the fact that I'm running this code on a I7 processor machine, and no matter how much I set the Pool to it's always running only two processes in parallel so is there any suggestion on how can I run 6 processes in parallel? such that the first three processes use L=6 and call compute_cluster with parameter values from 1:3 in parallel and at the same time the other three processes run the same function with the same parameter values but this time the Global L value is 7 ? any suggestions is highly appreciated

Answer 1

There are a few things wrong here. First, as to why you always only have 2 processes going at a time -- The reason is because range(1, 3) only returns 2 values. So you're only giving the pool 2 tasks to do before you close it.

The second issue is that you're relying on global state. In this case, the code probably works , but it's limiting your performance since it is the factor which is preventing you from using all your cores. I would parallelize the L loop rather than the "inner" range loop. Something like ¹ :

def wrapper(tup):
    l, r = tup
    # Even better would be to get rid of `L` and pass it to compute_cluster
    global L
    L = l
    compute_cluster(r)

for r in range(1, 3):
    p = Pool(6)
    p.map(wrapper, [(l, r) for l in range(6, 24)])
    p.close()

This works with the global L because each spawned process picks up its own copy of L -- It doesn't get shared between processes.

^{1 Untested code}

---

As pointed out in the comments, we can even pull the Pool out of the loop:

p = Pool(6)
p.map(wrapper, [(l, r) for l in range(6, 24) for r in range(1, 3)])
p.close()