RawArray not modified by processes as shared memory for Python multiprocessing

Question

I am working with python multiprocessing. Using Pool to start concurrent processes and RawArray to share an array between concurrent processes. I do not need to synchronize the accessing of RawArray, that is, the array can be modified by any processes at any time.

The test code for RawArray is: (do not mind the meaning of the program as it is just a test.)

from multiprocessing.sharedctypes import RawArray

import time

sieve = RawArray('i', (10 + 1)*[1])   # shared memory between processes

import multiprocessing as mp

def foo_pool(x):
        time.sleep(0.2)
        sieve[x] = x*x  # modify the shared memory array. seem not work ?    
        return x*x

result_list = []

def log_result(result):  
        result_list.append(result)

def apply_async_with_callback():
        pool = mp.Pool(processes = 4)    
        for i in range(10):
            pool.apply_async(foo_pool, args = (i,), callback = log_result)
        pool.close()
        pool.join()
        print(result_list)  

        for x in sieve:
            print (x)        # !!! sieve is [1, 1, ..., 1]

if __name__ == '__main__':
        apply_async_with_callback()

While the code did not work as expected. I commented the key statements. I have got stuck on this for a whole day. Any help or constructive advices would be very appreciated.

Answer 1

• time.sleep fails because you did not import time
• use sieve[x] = x*x to modify the array instead of sieve[x].value = x*x

• on Windows, your code creates a new sieve in each subprocess. You need to pass a reference to the shared array, for example like this:

   def foo_init(s): global sieve sieve = s def apply_async_with_callback(): pool = mp.Pool(processes = 4, initializer=foo_init, initargs=(sieve,)) if __name__ == '__main__': sieve = RawArray('i', (10 + 1)*[1])

Answer 2

You should use multithreading instead of multiprocessing, as threads can share memory of main process natively.

If you worry about python's GIL mechanism, maybe you can resort to the nogil of numba .

Answer 3

Working version:

from multiprocessing import Pool, RawArray
import time


def foo_pool(x):
    sieve[x] = x * x  # modify the shared memory array.


def foo_init(s):
    global sieve
    sieve = s


def apply_async_with_callback(loc_size):
    with Pool(processes=4, initializer=foo_init, initargs=(sieve,)) as pool:
        pool.map(foo_pool, range(loc_size))

    for x in sieve:
        print(x)


if __name__ == '__main__':
    size = 50
    sieve = RawArray('i', size * [1])  # shared memory between processes
    apply_async_with_callback(size)