Using python multiprocess pool to a function

Question

I have a function

def dist_to_center(ra_center,dec_center):
    # finding theta
    cos_ra = np.cos(ra_center-var1['ra'])
    cos_dec = np.cos(dec_center-var1['dec'])
    sin_dec = np.sin(dec_center)*np.sin(var1['dec'])

    theta = np.arccos((cos_ra*cos_dec)+sin_dec*(1-cos_ra))
    numerator = theta*comoving_dist
    denominator = 1+var1['zcosmo']

    # THE FINAL CALCULATED DISTANCE TO CENTRE
    dist_to_center = (numerator/denominator) 
    return dist_to_center

I want to make use of my processors, so I am using multiprocess pool like this:

if __name__ == '__main__':
    pool = Pool(processes=6)
    pool.map(dist_to_center, ra_center, dec_center) #calling the function with it's inputs
    pool.close()
    pool.join()

The code seems to be proper and is working, but only 1 processor is running instead of the 6 I have called. What am I doing wrong here?

Answer 1

You are passing a pair of one-dimensional arrays to the Pool. You need to slice the arrays yourself to make the Pool understand how to process them efficiently. For example:

def dist_to_center_mapper(arrays):
    return dist_to_center(arrays[0], arrays[1])

ra = np.split(ra_center, 6)
dec = np.split(dec_center, 6)
pool = Pool(processes=6)
pool.map(dist_to_center_mapper, zip(ra, dec)) 

I think the "mapper" function is required because Pool.map() takes only a single iterable of arguments. So we zip together the two lists of array slices so they get doled out together to the multiple processes. Note that you could split the arrays into more pieces than the number of processes if you want, if some pieces may take different amounts of time etc.