在大型矩阵的所有行对上进行计算的有效方法？

Question

I need to do some time-consuming calculations on all row-pairs of a large matrix M, like:

for i in range(n):
    for j in range(i+1,n):
        time_comsuming_calculation(M[i,:],M[j:])

Since I am new to parallel computing , after studied the example in Writing parallel computation results in shared memory , I tried to do parallel computing with joblib as below:

dump(M, M_name)
M=load(M_name,mmap_mode='r')
...
Parallel(n_jobs=num_cores)(delayed(paracalc)(u,v,M)
                                for u,v in itertools.combinations(range(M.shape[0]),2))

However, it turned to be unbearably much slower than non-parallel version. Computing on each row-pair consumed even more seconds than num_cores=1 . I am wondering what's wrong with my parallel implementation. Is mpi4py a better choice? Any suggestions will be appreciated.

Answer 1

Okay, still no answers but I've managed to work it out. The first interesting fact I found is when I commented out these two lines,

# dump(M, M_name)
# M=load(M_name,mmap_mode='r')

by which the memmap array were no longer used to take place of memory array, it went much faster. I don't know why up to now. Is there a memmap lock or something?

Then, I read this article Parallel and HPC with Python (or numpy) and decided to turn to mpi4py . After hours of struggling with debugging, I got satisfying results.