Is it possible to force multiple python processes to share the same memory via a C++ extension?

Question

Here is my attempt:

int* globalvar = new int[8];
void cpp_init(){
    for (int i = 0; i < 8; i++)
        globalvar[i] = 0;
}
void writeAtIndex(int index, int value){
    globalvar[index] = value;
}
int accessIndex(int index){
    return globalvar[index];
}
BOOST_PYTHON_MODULE(MpUtils){
    def("cpp_init", &cpp_init);
    def("writeAtIndex", &writeAtIndex);
    def("accessIndex", &accessIndex);
}

and in the python file

def do_stuff(index):
    writeAtIndex(index, randint(1, 100))
    time.sleep(index/10)
    print([accessIndex(i) for i in range(8)])

if __name__ == '__main__':
    freeze_support()
    processes = []
    cpp_init()
    for i in range(0, 10):
        processes.append( Process( target=do_stuff, args=(i,) ) )
    for process in processes:
        process.start()
    for process in processes:
        process.join()

And the output is this:

[48, 0, 0, 0, 0, 0, 0, 0]
[0, 23, 0, 0, 0, 0, 0, 0]
[0, 0, 88, 0, 0, 0, 0, 0]
[0, 0, 0, 9, 0, 0, 0, 0]
[0, 0, 0, 0, 18, 0, 0, 0]
[0, 0, 0, 0, 0, 59, 0, 0]
[0, 0, 0, 0, 0, 0, 12, 0]
[0, 0, 0, 0, 0, 0, 0, 26]

Could someone explain why this is not working? I tried printing the globalvar and it's always the same value. There shouldn't be any race conditions as wating for 0.1 to 0.8 seconds should be more than enough to make the computer write something. Shouldn't C++ be accessing directly the location of the pointer?

Thanks

Answer 1

Processes can generally access only their own memory space. You can possibly use the shared_memory module of multiprocessing to share the same array across processes. See example in the linked page.

Answer 2

The following loop will create a list of processes with consecutive first args , from 0 to 9 . (First process has 0 , second one 1 and so on.)

    for i in range(0, 10):
        processes.append( Process( target=do_stuff, args=(i,) ) )

The writing may take a list of numbers, but the call to writeAtIndex will be called with the first element only:

def do_stuff(index):
    writeAtIndex(index, randint(1, 100))

Function writeAtIndex takes one int , not a list:

void writeAtIndex(int index, int value)

That's why your output has a value on each position.

And as said by @Voo and in the docs :

The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads.

Process objects represent activity that is run in a separate process.

So there is a globalvar for each process.

If you want to share data between threads you can use the threading module. There are details you should read though:

due to the Global Interpreter Lock, only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation). If you want your application to make better use of the computational resources of multi-core machines, you are advised to use multiprocessing or concurrent.futures.ProcessPoolExecutor.