Passing Keys and Values of a Dictionary to a Function in Parallel?

Question

I am trying to pass the keys and values of a dictionary to a function in parallel in Python using the multiprocessing module.

I am running a series of several thousand calculations in parallel, the main function initially takes an array that describe molecular coordinates in cartesian form and then does some stuff with it. Just mapping a list of arrays through a list comprehension to the function using multiprocessing pool worked fine.

def main(grid_point):
    do stuff...

if __name__ == "__main__":
    grid_points = [] # some list of arrays
    run_pool = multiprocessing.Pool()
    run_pool.map(main, [grid_point for grid_point in grid_points])
# Works fine

However, I wish to store some data that is specific to each molecular coordinate such that the order in which the values are stored correspond to their initial indexes in the list prior to the parallel calculation. Note, each calculation will not take the same time to run and thus I may jumble the data.

In order to overcome this, I wish to pass each key: value pair of a dictionary to the function in parallel. Now the key will correspond to the original index prior to the parallel run and thus I have a way to ensure my data does not get scrambled.

def main(gp_key, grid_point):
    do stuff...

if __name__ == "__main__":
    grid_points = [] # some list of arrays
    grid_points_map = {k: v for k, v in enumerate(grid_points)} # Dict of indexes and arrays
    run_pool = multiprocessing.Pool()
    run_pool.map(main, {gp_key: grid_point for gp_key, grid_point in grid_points_map})
# Does not work

Any insights into how best to go about passing the key: value pairs in parallel would be a great help, or even any suggestions for how to ensure the data stored at the end corresponds to its original index. The only other way I can think of doing it is passing a tuple to the function using pool ie run_pool.map(main, [(k, v) for k, v in enumerate(grid_points)] and then unpacking the tuple into an index and array in main() as index, array = grid_point .

Answer 1

You are using run_pool.map() incorrectly. When you do run_pool.map(main, ...) , it will pass each element from the iterable to the function main() one at a time. In the case of a dictionary, the elements are the keys. So you are passing the values 0 , 1 , 2 , etc. to main() but not the grid_point value. Instead, you want to use starmap() directly on the result from enumerate() . The intermediate dictionary is not necessary.