在Python中使用多个CPU实现非常快速的功能

Question

What I have is a function that can be looped 10,000 times in 1 second. However, I need to perform this function tens to hundreds of millions of times. As expected due to the overhead of using CPUs, using the native multiprocessing package with my 4 cores slows the 10k loop to 1.5 seconds. Using the chunksize parameter in multiprocessing helped trivially. Is there any way to get multiple processes to call this function with a speedup greater than the overhead?

A truncated version of the function:

rands = np.random.random((200, 1000000))

def randfunc(i):
    Q = np.concatenate([rands[:,[i]], rands[:,[i]] * rands[:,[i+1]]],axis=1)
    Q2 = np.dot(np.transpose(Q),Q)
    Q3 = np.linalg.inv(Q2) * Q2[1,1]
    return Q3

Answer 1

I was able to cut the run time in half by using the ipyparallel package simply using map_sync instead of the multiprocessing package to parallelize the function. I'm not really sure why the former package has less overhead than the latter, but for the former, loading the data did take long, as opposed to the latter, which recognizes rands as a variable during parallel execution. However, in both cases the data are stored in RAM. If anyone reads this and knows the reason why ipyparallel is faster, do comment.