How to avoid copy of a non-global numpy array between process?

Question

I've read " Is shared readonly data copied to different processes for Python multiprocessing? " but the array described there is global. Is it possible to do the same with local arrays?

Answer 1

I don't think so - but you can save stuff to a module variable. If you do this before the fork (and you are not on windows) it should work fine.

Eg

import mymodule

def somefunc(parameter):
    # do something with mymodule.var

# load/process local data
# save to module variable
mymodule.var = var
# now fork
p = multiprocessing.Pool(8)
p.map(somefunc, list_of_params)

If you use ipython you need to put somefunc in a module too (pickling functions in main doesn't seem to work).