Parallellising a for loop for a function with multiple inputs and outputs

Question

I've read several posts on multiprocessing in Python, but it's not clear to me how can I use them for my problem (where I have multiple inputs and outputs). Most of the available examples consider a single output function with a rather simple structure.

Here is the code in Python:

import numpy as np

n = 1000
i1 = np.random.random(n)
i2 = np.random.random(n)
i3 = np.random.random(n)
i4 = np.random.random(n)

o1 = np.zeros(n)
o2 = np.zeros(n)
o3 = np.zeros(n)

def fun(i1,i2,i3,i4):
    o1 = i1 + i2 + i3 + i4
    o2 = i2*i3 - i1 + i4
    o3 = i1 - i2 + i3 + i4

    if o1 < o2:
        o1 = o2
    else:
        o2 = o1

    while o1 + o2 > o3:
        o3 = o3 + np.random.random()

    return o1,o2,o3

for i in range(n):  # I want to parallellise this loop
    o1[i],o2[i],o3[i] = fun(i1[i],i2[i],i3[i],i4[i])

I am only looking for a way to parallelize the for loop. How can I accomplish this?

Answer 1

I will use a generator to combine your input lists i1 to i4. Your math-function fun will return a list-object. Now I have a single argument as input (the generator) and get a single object as output (the list). I have tried the code below and it works.

You can add a sleep-command in your fun -function to see the speed-gain when using multiple processes. Otherwise your fun -function is too simple to really benefit from multi-processing.

import numpy as np
from multiprocessing import Pool

n = 1000
i1 = np.random.random(n)
i2 = np.random.random(n)
i3 = np.random.random(n)
i4 = np.random.random(n)

def fun(a):
    o1 = a[0] + a[1] + a[2] + a[3]
    o2 = a[1]*a[2] - a[0] + a[3]
    o3 = a[0] - a[1] + a[2] + a[3]

    if o1 < o2:
        o1 = o2
    else:
        o2 = o1

    while o1 + o2 > o3:
        o3 = o3 + np.random.random()

    return [o1,o2,o3]

# a generator that fills the Pool input
def conc(lim):
    n = 0
    while n < lim:
        yield [i1[n], i2[n], i3[n], i4[n]]
        n += 1


if __name__ == '__main__':
    numbers = conc(n)
    with Pool(5) as p:
        print(p.map(fun, numbers))