Python - retrieving a variable from memory?

Question

I have two python scripts that need to communicate large variables to one another: python1.py and python2.py.

Let's say python1.py is running and has created a list variable 'x' which is very large. At the moment, I am saving (pickling) 'x' to the hard drive and then using subprocess to run python2.py which then loads up 'x' from the hard drive (I need to have two different python files because I am trying to parallelize a computation).

Is there an alternative, where I can call python2.py with an argument which is a pointer to memory, and then have python2.py create 'x' based on directly looking it up in the memory?

Answer 1

If you are looking into splitting computation across processes, I would strongly recommend giving the "multiprocessing" module a read which has concepts like process pools, managers and ability to share high-level data structures across process boundaries. For eg take a look at "sharing state between two processes" section in the docs. From the docs:

from multiprocessing import Process, Array

def f(a):
    for i in range(len(a)):
        a[i] = -a[i]

if __name__ == '__main__':
    arr = Array('i', range(10))

    p = Process(target=f, args=(arr,))
    p.start()
    p.join()

    print(arr[:])

#output: [0, -1, -2, -3, -4, -5, -6, -7, -8, -9]