Communicating between Pre-Existing Python Processes using Multiprocessing

Question

Is it possible to use 'multiprocessing' communication to a process that I didn't spawn?

The 'multiprocessing' module looks ideal for my needs, but in all the examples one process spawns the other. How can I get a 'handle' on the process I want to communicate with if I didn't spawn it?

Background Information

There are three processes in total - all Python 2.7 - a worker, a watchdog, and a web server.

The worker is a long-running test system. I want the web server to be able to query the worker to see how it is progressing. As the worker may often crash, a watchdog process restarts it from time to time (and hence it will have a new PID).

I have access to the source of the worker and the watchdog, but I want to modify those as little as possible.

My idea is to have the web server query the worker periodically and share, say, a dict.

Answer 1

From what you've described, you want the "worker", which may crash and be restarted, to publish some dictionary of data to the "web server." I think the easiest way to do this is to simply have the web server accept incoming messages from the worker via HTTP PUT. You can encode the worker's dict as JSON and put that in the HTTP body. This makes use of the protocols you are likely already using, and avoids having to coordinate restarts of the worker to rediscover it when its PID changes.

Of course, if the server hasn't seen an update after a certain amount of time, it can consider the worker to be dead. It may even be able to subsume the watchdog functionality therefore.

Answer 2

Is it possible to use 'multiprocessing' communication to a process that I didn't spawn?

No. The multiprocessing module can work the way it does because the processes 'know' their relation 'behind the scenes' even though you don't see it.

There are a couple of possible solutions;

1. Fold the watchdog into the web server. Then you can use multiprocessing to launch the worker and communicate between the worker and web server.
2. Use IPC, eg sockets . Make the webserver listen on a seperate port for connections from the worker. Using a host and port combination, socket.getaddrinfo() will return the info needed for a connection.
3. Use a specially crafted HTTP PUT request to send data from the worker to the web server as John Zwick explained.