Nested for loop using multiprocessing.pool

Question

I have to use a nested for loop for a particular problem when the iterators are run and time for each run. I am trying to carry it out for different values of a hyperparameter(here T). I am trying to parallelise this process(3 processes) using multiprocessing.pool method. But I am not able to figure out how to implement it.

def simulate(T,runs,time,param1, param2, param3, param4):
for i in tqdm(range(runs)):
    #Reset parameters
    for j in range(time):
        #Do some mathematics
#Some more mathematics
return (some output)

As it can be seen that the number of parameters for the function are many. So also I am not sure how to incorporate in functools.partial. Any guidelines?

Answer 1

If I understand you correctly, you want to run the simulate() method with different values of T and compare the results. To implement this using multiprocessing, you just need to set up a Pool with the right number of processes, and use map to run your function on a list of values of T . You also need to use partial to turn your function from one that takes seven arguments into a function that only needs one, with constant values for the other six. This is important because map needs to know which argument is the one that is being varied. Here's a (nontested) example:

import multiprocessing as mp
from functools import partial

# Create pool with desired number of processes
pool = mp.Pool( processes=3 )
# Make a partial function with preset values for params
partial_function = partial( simulate, runs=runs, 
   time=time, param1=param1, param2=param2, 
   param3=param3, param4=param4 )
# Dummy values for what T should be
options_for_T = [100, 200, 300, 400, 500, 600]
# Results will be a list of the output of simulate() for each T
results = pool.map( partial_function, options_for_T )

EDIT: I should also point out that using tqdm here might be counterproductive, since all your processes will be talking over each other