multiprocessing map python speed issue with list

Question

I have two similar piece of code, speed is very different:

Code 1: Execution in 16seconds

 def mc05():
     num_procs = 4
     iters = 1000000000
     its = iters / num_procs  
     pool = mp.Pool(processes=num_procs)

     result = pool.map(mpf.integrate3, [its] * num_procs)    

     print(sum(result) / float(num_procs))
     pool.terminate();   pool.join();

Code2: Time execution: 32seconds (=1 process)

 def mc05():
     num_procs = 4
     iters = 1000000000
     its = iters / num_procs  
     pool = mp.Pool(processes=num_procs)

     result = pool.map(mpf.integrate3, [iters] )    

     print(sum(result) / float(num_procs))
     pool.terminate();   pool.join();   

The only difference comes from the list where in the code 2, it has been simplified.... Why this is different ?

Answer 1

pool = mp.Pool(processes=num_procs)

tells multiprocessing, how many concurrent processes to run, which is 4 in your case. But 4 processes will only be started if your input list is larger than 4.

Now when you do

pool.map(mpf.integrate3, [iters] )

your [iters] list has only one value ie [1000000000] so only one process will be started.

When you do

result = pool.map(mpf.integrate3, [its] * num_procs)

your list [its] * num_procs has 4 values ie [250000000, 250000000, 250000000, 250000000] , so 4 processes will be started by pool.

Answer 2

Code1

... [its] * num_procs == [its, its, its, its]

Code2

... [iters] == [1000000000]

The map function iterates through the list. In Code2 it is only iterating through one value. Code1 is iterating through a list of size 4 calling that function for every item in the list. You should check if you are getting the same results for both pieces of code. I'm not too familiar with mpf.integrate3, but code1 will be slower the map function is running through the list of size 4). Either that or Code2 is slower because only one process is running on the list [1000000000]. I'm not quite sure how all of it is working, because I'm not that familiar with the multiprocessing pool.