transferring data between processes in python

Question

I'm not new to python but still looking for the best appropriate way of sharing string-formatted data between processes.

Currently I have 4 processes (1 parent + 3 Childs in multiprocessing), parent is fetching data from long poll server and after few checks sends data to mysql server.

Then Childs (1 for each kind of tasks) are processing the data in the way I need to.

Everything works fine for now, my data is securely stored and always in access, especially if I need to debug or implement new features, but after all it works slower with project growth (as for web app - not good). As you might assume, kind of data stored in db is list or its objects. I know that there are some problems in python with transferring data between processes ('d be better to say, software restrictions)...

I was thinking about temporary storing data in JSON or simple txt files, but concurrency wouldn't allow me to do that. Also I could try using sockets, but is it worth to start callback server for such a purpose?

Asynchronous thing didn't work for me either. So what are my options in this case? I don't need to loose any piece of data, just as I need to keep it working fast. Any other measures are welcome :)

ps most of related topics are outdated or didn't answer my question, because I'm sure that the way I'm working with data isn't best so far.

Answer 1

There're many IPC ways(socket, shared memory, pipe, FIFO...), I think you can try pipe , it is simple. see the example below:

import os, sys


def child(prefix, r, w):
    os.close(w)
    print(prefix + ':read')
    s = os.read(r, 1024)
    print(prefix + ':got:' + s)
    print(prefix + ':close')
    os.close(r)
    print(prefix + ':exit')
    sys.exit(0)


r0, w0 = os.pipe()
r1, w1 = os.pipe()
r2, w2 = os.pipe()

if os.fork():
    if os.fork():
        if os.fork():
            os.close(r0)
            os.close(r1)
            os.close(r2)
            print('p:write')
            os.write(w0, 'hello world')
            os.write(w1, 'hello world')
            os.write(w2, 'hello world')
            print('p:close')
            os.close(w0)
            os.close(w1)
            os.close(w2)
            print('p:exit')
            sys.exit(0)
        else:
            child('c0', r0, w0)
    else:
        child('c1', r1, w1)
else:
    child('c2', r2, w2)

Answer 2

The multiprocessing module provides some tools to do this: https://docs.python.org/3.6/library/multiprocessing.html#exchanging-objects-between-processes

Specifically take a look at Queue . Any process can push messages onto or pop messages from the queue. So for example, a parent could push tasks, the children wait for tasks and get them on a first-come-first-served basis. If one child is busy working on that task, the next child will get it.

For example, if you needed your child processes to work on data fetched by your parent process, you could have something like this:

from multiprocessing import Process, Queue

def do_thing(q):
    data = q.get()  # Get data queued by the parent process
    # do something with data

if __name__ == '__main__':
    q = Queue()
    p = Process(target=do_thing, args=(q,))
    p.start()
    while True:
        data = get_data_from_server()
        q.put(data)  # Queue data for a child server

If you want something a bit more basic, you could use a Pipe , which allows two-way data transfer between processes.