python多处理在不同的python进程之间共享数据

Question

Multiprocessing allows me to share data between processes started from within the same python runtime interpreter. But what if i had a need to share data between processes started by separate python runtime processes? I was looking at multiprocessing.Manager which seems to be the right construct for it. If I create a manager i can see its address:

>>> from multiprocessing import Manager
>>> m=Manager()
>>> m.address
'/tmp/pymp-o2TCd_/listener-Qld03B'

And the socket is there:

adrian@sammy ~/temp $ netstat -naA unix | grep pymp
unix  2      [ ACC ]     STREAM     LISTENING     1220401  /tmp/pymp-     o2TCd_/listener-Qld03B

If I start a new process with multiprocessing.Process it spawns a new python interpreter that somehow inherits information about these shared constructs like this Manager. Is there a way to access it from a new python process NOT spawned from the same one that created the Manager?

Answer 1

You are on the (or a) right track with this.

In a comment, stovfl suggests looking at the remote manager section of the Python multiprocessing Manager documentation ( Python2 , Python3 ). As you have observed, each manager has a name-able entity (a socket in /tmp in this case) through which each Python process can connect to a peer Python process. Because these are accessible from any process, however, they each have an access key.

The default key for each Manager is the one for the "main process", and it is a string of 32 random bytes :

class _MainProcess(BaseProcess):

    def __init__(self):
        self._identity = ()
        self._name = 'MainProcess'
        self._parent_pid = None
        self._popen = None
        self._config = {'authkey': AuthenticationString(os.urandom(32)),
                        'semprefix': '/mp'}
        # Note that some versions of FreeBSD only allow named
        # semaphores to have names of up to 14 characters.  Therefore
        # we choose a short prefix.
        #
        # On MacOSX in a sandbox it may be necessary to use a
        # different prefix -- see #19478.
        #
        # Everything in self._config will be inherited by descendant
        # processes.

but you may assign your own key, which you can then know and therefore use from anywhere else.

There are other ways to handle this. For instance, you can use XML RPC to export callable functions from one Python process, callable from anything—not just Python—that can speak XML RPC. See the Python2 or Python3 documentation. Heed this warning (this is the py3k variant but it applies in py2k as well):

Warning : The xmlrpc.client module is not secure against maliciously constructed data. If you need to parse untrusted or unauthenticated data see XML vulnerabilities.

Do not, however, assume that using a multiprocessing.Manager instead of XML RPC secures you against maliciously constructed data. Those are just as vulnerable since they will unpickle arbitrary data. See Attacking Python's pickle for more about this.