How to efficienctly convert numpy array to shared memory object on 64 bit Python on Windows

Question

I have a large numpy array which I'd like to convert to a shared memory object in order to process it simultaneously using multiple processes.

For example, the numpy array can be created using:

M=rand(1000000000) 

This takes about 15 seconds and uses about 8GB of memory. I use the following code to convert M to a shared memory object:

from multiprocessing.sharedctypes import RawArray
sys.maxint=2**64
N=RawArray("d",M)

This takes 18 minutes and consumes an additional 46GB of memory during the conversion process (after conversion is complete memory consumption is still at 8GB).

Both the long running time and high memory consumption of the conversion process make it unsuitable for me to use it efficiently in multi-processing.

Any efficient solution out there?

Answer 1

I've been days trying to find a solution same as yours, what I came out with is that if not impossible it is not possible the way it would be on a unix system just following the multiprocessing manual. On my case it only works sharing a int. Arrays just don't get updated when i modify them between processes.