Python multiprocessing pool inside a loop

Question

I have a loop and in each iteration, there are tasks that should be executed in parallel. I need to wait for the tasks to run in parallel in the current iteration and then go to the next iteration.
For example,

a = [1,2,3,4,5,6,7,8,9,10]
pool = Pool(mp.cpu_count())

def fun1(x):
   ....

def fun2(x):
   ....

def fun3(x):
   ....

for x in a:
  pool.map(fun1, x)
  pool.map(fun2,x)
  pool.map(fun3,x)
  pool.close()
  pool.join()

Is this the right way? Or how do I achieve this?

Answer 1

Based on your comment, you would like to run fun1, fun2, fun3 in parallel for x=1 , wait until they all finish, then move on to x=2 and repeat. This can be achieved like this:

import multiprocessing as mp
a = [1,2,3,4,5,6,7,8,9,10]

def fun1(x):
   ....

def fun2(x):
   ....

def fun3(x):
   ....

for x in a:
    # Create separate process for each function 
    p1 = mp.Process(target=fun1, args=(x))
    p2 = mp.Process(target=fun2, args=(x))
    p3 = mp.Process(target=fun3, args=(x))
    # Start all processes 
    p1.start()
    p2.start()
    p3.start()
    # Wait till they all finish and close them 
    p1.join()
    p2.join()
    p3.join()

Alternatively, if you would like to run fun1 for all x in a , the run fun2 then fun3 , you can use a multiprocessing pool instead:

import multiprocessing as mp
a = [1,2,3,4,5,6,7,8,9,10]
pool = mp.Pool(mp.cpu_count())

def fun1(x):
   ....

def fun2(x):
   ....

def fun3(x):
   ....

# Run fun1 for all values in a
pool.map(fun1, a)
# Run fun2 for all values in a
pool.map(fun2, a)
# Run fun3 for all values in a
pool.map(fun3, a)
# Close pool 
pool.close()
pool.join()

In the multiprocessing pool case, pool.map(fun2, a) will not run unless pool.map(fun1, a) finishes running. For more information on Python's multiprocessing module, I highly recommend reading the documentation