Trying to use multiprocessing to fill an array in python

Question

I have a code like this

x = 3;
y = 3;
z = 10;
ar = np.zeros((x,y,z))

from multiprocessing import Process, Pool

para = []
process = []
def local_func(section):
    print "section %s" % str(section)
    ar[2,2,section] = 255
    print "value set %d", ar[2,2,section]

pool = Pool(1)

run_list = range(0,10)
list_of_results = pool.map(local_func, run_list)

print ar

The value in ar was not changed with multithreading, what might be wrong?

thanks

Answer 1

You're using multiple processes here, not multiple threads. Because of that, each instance of local_func gets its own separate copy of ar . You can use a custom Manager to create a shared numpy array, which you can pass to each child process and get the results you expect:

import numpy as np
from functools import partial
from multiprocessing import Process, Pool
import multiprocessing.managers

x = 3;
y = 3;
z = 10; 

class MyManager(multiprocessing.managers.BaseManager):
    pass
MyManager.register('np_zeros', np.zeros, multiprocessing.managers.ArrayProxy)


para = []
process = []
def local_func(ar, section):
    print "section %s" % str(section)
    ar[2,2,section] = 255 
    print "value set %d", ar[2,2,section]

if __name__ == "__main__":
    m = MyManager()
    m.start()
    ar = m.np_zeros((x,y,z))

    pool = Pool(1)

    run_list = range(0,10)
    func = partial(local_func, ar)
    list_of_results = pool.map(func, run_list)

    print ar

Answer 2

Well, multi-threading and multi-processing are different things.

With multi-threading threads share access to the same array.

With multi-processing each process has its own copy of the array.

Answer 3

multiprocessing.Pool is a process pool, not a thread pool.

If you want thread pool, use multiprocess.pool.ThreadPool :

---

Replace:

from multiprocessing import Pool

with:

from multiprocessing.pool import ThreadPool as Pool