使用python多处理同时写入float数组

Question

I coded a matrix multiplier a while ago, in an attempt to make it faster I tried to make it threaded just to discover that threads run on the same process.. I later discovered the multiprocessing library which I have implemented in the code below. Now, I don't know how to merge the work made by the processes spawned since the result is not in shared memory.

How can I merge the distributed calculations into the "final_multi" variable?

Heres my code:

#!/usr/bin/env python
import numpy as np
from multiprocessing import Process, Array

T=64
v1 = np.empty([T,T], dtype=np.float32)
v2 = np.empty_like(v1)
final_multi = np.empty_like(v1)
#shared = Array('f', final_multi) This doesnt work
def calclinea(mat1, mat2, fil, col):
    escalar = 0
    for vl in range(T):
        escalar += mat1[fil,vl]*mat2[vl,col]
    return escalar

def mulshared(vec1, vec2, froY, toY, froX, toX):
    global final_multi
    for y in range(froY,toY):
        for x in range(froX, toX):
            final_multi[x,y] = calclinea(vec1,vec2,x,y)
            #shared[x,y] = calclinea(vec1,vec2,x,y)

def main():
    for r in range(T):    ### Allocate host memory
        for c in range(T):
            v1[r,c] = r
            v2[r,c] = c+2
            final_multi[r,c] = 0

    #p1 =Process(target=mulshared, args=(v1,v1,0,(T*1/4 -1),0,T))
    #p2 =Process(target=mulshared, args=(v1,v1,(T*1/4),(T*2/4 -1),0,T))
    #p3 =Process(target=mulshared, args=(v1,v1,(T*2/4),(T*3/4 -1),0,T))
    p4 =Process(target=mulshared, args=(v1,v1,T*3/4,T*4/4,0,T)) #All four processes to demo distribution of data, only 4th is initialized so result can be seen, p1 result is all zeros so..

    p4.start()
    p4.join()

    print "\nfinal_multi\n", final_multi
main()

I know this is a inefficient way of matrix multiplication, I just want to learn how multiprocessing works, Thanks in advance.

Answer 1

You can use the sharedmem module, it's an enhanced version of the multiprocessing module that comes with Python. It offers a nice an easy way to share memory between processes.

import sharedmem as shmem

out_matrix = shmem.empty((400,400))

def do_work(x):
    out_matrix[100*x:100*(x+1), :] = x

def main():
    with shmem.MapReduce(np=4) as pool:
        pool.map(do_work, range(4))

In this minimal example, the output matrix will be filled by four workers in parallel.