How to pass multiple arguments in Multiprocessing executor.map() function

Question

I have been watching several videos on Multiprocessing map function.

I know that I can send one list as an argument to the function I want to target with Multiprocessing, and that will call the same function n times (dependent upon the size of that passed list).

What I am struggling to do is what if I want to pass multiple arguments to that function?

I basically have a List whose size is n (it can vary but for current case, its 209)

My function requires 3 arguments...

1. the index of the list (0, 1, 2 etc.)
2. Another list containing data
3. A fixed integer value

I could have used the 2nd and 3rd arguments as global variables, but that doesn't work for me because I have to call the map function in a while loop... and in every another iteration, the values of these two will change.

My function returns two values which I need to access in the function from where it was called. This is what I have tried but it didn't work for me,

def main_fun():
    with concurrent.futures.ProcessPoolExecutor() as executor: 

        results = executor.map(MyFun, (row, pop[0].data, fitness) for row in range(0, len(pop[0].data)))

        for result in results:
            print(result)

I also tried to use ZIP function but again, with no success.

Answer 1

If your second and third arguments to your worker function (ie the first argument to map ), then you can use method functools.partial to have the second and third arguments specified without resorting to the use of global variables. If your worker functions is, for example, foo , then:

from concurrent.futures import ProcessPoolExecutor
from functools import partial


def foo(idx: int, lst: list, int_value: int):
    ...

def main():
    with ProcessPoolExecutor() as executor:
        worker = partial(foo, lst=pop[0].data, int_value=fitness)
        executor.map(worker, range(0, len(pop[0].data)))

if __name__ == '__main__':
    main()

So now we only have to pass to map function worker , which will be called two fixed arguments, and a single iterable argument.

If you are executing the map call in a loop, you will, of course, create a new worker functions by passing to functools.partial new arguments.

For example:

from concurrent.futures import ProcessPoolExecutor
from functools import partial


def foo(idx: int, lst: list, int_value: int):
    print(idx, lst[idx] * int_value, flush=True)

def main():
    l = [3, 5, 7]
    fitness = 9
    with ProcessPoolExecutor() as executor:
        worker = partial(foo, lst=l, int_value=fitness)
        executor.map(worker, range(0, len(l)))

if __name__ == '__main__':
    main()

Prints:

0 27
1 45
2 63