can I share a multiprocessing.Manager() instance across 3 processes?

Question

I'm trying to understand if I'm allowed to do the following in python:

• create a manager instance in my program
• fork the existing process N times
• in my initial process, send data to the manager (to some shared variable)
• in my forked processes, read data from the manager (from the shared variable)

Based on my understanding, it should be the main use-case for managers. However, I am trying this for more than 2 processes I'm getting pickle errors sometimes when trying to do this. With only 2 processes it always works My question is thus: is this supposed to work or not?

I wonder if maybe the Manager system is not based on a file-descriptor/socket so it's only made to communicate 1-to-1, or at least by default

Answer 1

No, you can't and shouldn't share a manager instance. The documentation says "A manager object controls a server process which manages shared objects. Other processes can access the shared objects by using proxies."

The multiprocessing.Manager provides a RPC(remote procedure call) feature, similar to Java RMI and MS COM. So you should distinguish a server process from client processes. You need to understand the APIs such as start() , serve_forever() , connect() , etc. You don't need to call the low level API fork() .

This is an example of a RPC server and client.

import sys
from multiprocessing.managers import BaseManager

class MathsClass:
    def add(self, x, y):
        return x + y

class MyManager(BaseManager):
    pass

if __name__ == '__main__':
    mode = sys.argv[1]
    if mode == 'server':
        MyManager.register('Maths', MathsClass)
        manager = MyManager(address=('localhost', 3000), authkey=b'')
        manager.get_server().serve_forever()
    elif mode == 'client':
        MyManager.register('Maths', MathsClass)
        manager = MyManager(address=('localhost', 3000), authkey=b'')
        manager.connect()
        maths = manager.Maths()
        print(maths.add(4, 3))

You can run a server and client by specifying the 'server' and 'client' command line argument respectively. It uses a TCP socket but other channels can be used such as a unix socket or a WIN32 named pipe.

As a side note, I'll explain how an official sample works. I chose it because it's somewhat confusing in my opinion. The following is one of the official samples, edited for brevity.

from multiprocessing.managers import BaseManager

class MathsClass:
    def add(self, x, y):
        return x + y

class MyManager(BaseManager):
    pass

MyManager.register('Maths', MathsClass)

if __name__ == '__main__':
    with MyManager() as manager:
        maths = manager.Maths()
        print(maths.add(4, 3))

In the with MyManager() as manager: line, manager.start() is called via a context manager. This creates a server process as a child process. The server waits for a remote call request from a client. Now the main process as a remote call client calls the maths.add() , where the maths is a proxy object. The maths.add(4, 3) call will serialize the method name and arguments into a remote call request, and send the request to the server(the child process). The server will parse the request, execute the actual implementation of the MathsClass.add() , serialize the return value into a remote call response, and send the response to the client(the main process).