How can I make many perallel processes make changes to a single shared NumPy array?

Question

I have scoured the internet for an answer, and nothing I can find applies to my situation. I have read about multiprocessing.Manager , have tried passing things back and forth, and none of it seems to play well with NumPy arrays.I ahve tried using Pool instead, but my target method does not return anything, it just makes changes to an array, so I wasn't sure how to set that up either.

Right Now I have:

def Multiprocess(self, sigmaI, sigmaX):
    cpus = mp.cpu_count()
    print('Number of cpu\'s to process WM: %d' % cpus)

    processes = [mp.Process(target = self.CreateMatrixMp, args = (sigmaI, sigmaX, i,)) for i in range(self.numPixels)]
    for p in processes:
        p.start()
    for p in processes:
        p.join()

The target function, CreateMatrixMp , takes the values passed, and after doing calculations, appends a value to an array data . This array is declared as self.data = numpy.zeros(self.size, numpy.float64) . If the details of the CreateMatrixMp method would help, I can post that as well.

I tried adding this above where the processes are run:

mgr = mp.Manager()
sharedData = mgr.Array(ctypes.c_numpy.float64, self.data)

and then passing sharedData to CreateMatrixMp , where it can be modified. Once all the processes have run and the array is complete, I simply do self.data = sharedData .

But this doesn't work (though I know I am not setting it up correctly). How should this be done with a NumPy array? I want each and every process (there will be thousands of them) to append to the same array.

Any help is enormously appreciated.

Answer 1

Welcome to the dark world of multiple threads. I think your big problem here is the mgr.Array puts synchronisation around the array. If you generate data quickly this will be a bottle-neck since processes will be waiting for the last to finish with the array. It is more efficient and will help if each process keeps a private copy of the nump array. Once you have fed in all the data then wait for all the processes to complete. Then you can combine all the arrays into self.data. This way none of the processed need wait on a shared resource. Neither this solution, nor yours, guarantee the order of the output list. I suspect from self.numPixels that order may be important. Many solutions, but the easiest is to feed in order with the data and do a self.data.sort(...) after all is done. Alternatively and faster, pre-create self.data and have the processes poke results in the correct location. self.data does not need to be a shared data structure since the processes are never changing anything in common. This works if arrays map to C-like arrays. It will not work for linked lists, etc. Hope this helps. Ask if you want more details.