Python Multiple function execution in Parallel

Question

Looking for a way to execute multiple functions in parallel. I had a look at multiprocessing, but wanted to see if anyone had an example for this use case:

I have the following:

data = function1(arg1,arg2,arg3,arg4)
data1 = functiun2(arg1,arg2,arg3,arg4, arg5, arg6)
data2 = functiun1(arg1,arg2,arg3,arg4, arg5)

# and so on for around 30 functions (only executes the 3 functions with separate args).

How would I bring these into multiprocessing threads?

Answer 1

I should be able to do something like this

from multiprocessing import Pool

# Define function 1

# Define function 2

if __name__ == '__main__':
    processes = 30

    pool1 = Pool(processes)
    result1 = pool1.starmap_async(function1, [[arg1,arg2,arg3,arg4] for i in range(processes)])

    pool2 = Pool(processes)
    result2 = pool2.starmap_async(function2, [[arg1,arg2,arg3,arg4, arg5, arg6] for i in range(processes)])

    pool3 = Pool(processes)
    result3 = pool3.starmap_async(function1, [[arg1,arg2,arg3,arg4, arg5] for i in range(processes)])

    data = result1.get()
    data2 = result2.get()
    data3 = result3.get()

    pool1.close()
    pool1.join()

    pool2.close()
    pool2.join()

    pool3.close()
    pool3.join()

It will execute each of your 3 functions 30 times. Note that I recommend condensing this with a loop, but I am leaving it as is for readability.