通过Python中的套接字在两个进程之间传递共享内存对象

Question

My application consists of two Python processes connected by the means of a TCP socket. The first process is designed to read files and put them in shared memory rather than push down the socket, so that the other process could read them.

import os
import zmq
import mmap
import time
import ctypes
import pickle
import dill
import tempfile
import multiprocessing as mp

MEETING_POINT = 'tcp://127.0.0.1:12345'
FILEFD, FILEPATH = tempfile.mkstemp()


def run_server():
    Server().run()


def read_file():
    return open(FILEPATH, 'r').read()


class Client(object):
    def __init__(self):
        self.ctx = zmq.Context()
        self.socket = self.ctx.socket(zmq.DEALER)
        self.socket.connect(MEETING_POINT)

    def cleanup(self):
        self.socket.close()
        self.ctx.term()

    def run(self):
        self.socket.send(dill.dumps(read_file()))


class Server(object):
    def __init__(self):
        self.ctx = zmq.Context()
        self.socket = self.ctx.socket(zmq.DEALER)
        self.socket.bind(MEETING_POINT)

    def cleanup(self):
        self.socket.close()
        self.ctx.term()

    def run(self):
        f = dill.loads(self.socket.recv())
        print(f)


def main():
    with open(FILEPATH, 'w') as fd:
        fd.write('blah')

    mp.Process(target=run_server).start()
    time.sleep(.5)  # a (poor, but tolerable) alternative to handshaking
    Client().run()


if __name__ == '__main__':
    main()

My question boils down to this: how do I pass information about the segment of memory, that the second process is to access, through the socket? I've tried the following (with both pickle and dill ), all to no avail:

• use mmap.mmap on a file directly or to store its contents. Result: TypeError: Required argument 'fileno' (pos 1) not found when unpickling on the server side.

• use multiprocessing.Array:

  with open(FILEPATH, 'r') as fd: contents = bytearray(fd.read()) arr = mp.Array(ctypes.c_byte, contents, lock=False)

Result: pickle.PicklingError: Can't pickle <class 'multiprocessing.sharedctypes.c_byte_Array_4'>: it's not found as multiprocessing.sharedctypes.c_byte_Array_4 when pickling on the client side.

I am convinced there's a simpler (and working) way to do this, is there? I can't pre-read all the necessary files and pack them into a multiprocessing.Array to pass as args when creating a server process, of course that would (supposedly) solve everything.

Answer 1

Answering my own question because I figured what exactly I were to do. According to man mmap , for the second process to be able to read the first one's memory it is sufficient to create a memory segment with MAP_SHARED flag. Therefore, as in mmap.mmap 's constructor this flag is set by default, all one has to do is pass the file name and not a class instance:

# sometime during the first process' execution
fd = open(filepath_to_process, 'w')
memory = mmap.mmap(fd.fileno(), 0)
socket.send(filepath_to_process)

# on the other side of socket, this happens:
unprocessed_filepath = socket.recv()
fresh_fd = open(unprocessed_filepath, 'r')
mm = mmap.mmap(fresh_fd.fileno(), 0)

# voilà! All the changes are now propagated to the second process

Of course, access modes for a file and a memory region have to match, but there's a separate section in man mmap for that, called "ERRORS" .

As for the shared memory, there are posix_ipc and sysv_ipc modules (the latter supports a wider variety of OSes).