How to properly terminate child processes with multiprocessing in python

Question

I have a few callback functions and I'd like to launch as multiple processes and have them all terminate via signal from the parent process.

My current way of doing this is creating a shared c_bool with multiprocessing.Value and setting it to True , then distributing it to all of my processes when they are created. My processes all run a while loop using the shared bool like so:

while myC_bool: ...keep running...

I can then just switch the bool to False from my parent process and all child processes will complete their final loop and exit.

I've been told by many people, and have read in the docs that one should try avoid using shared memory when using multiprocessing. I was told the best way to avoid this is to daemonize the process, give it a custom signal handler and send it a sigint/sigterm/etc...

My question is, is exclusively using the bool to keep a loop alive and ONLY ever alter it's value from my parent process, and read it from multiple child processes a suitable solution to make all of my child processes terminate quickly and safely? I feel like there is less overhead for all the children to just look at the one shared bool, than to send x number of sigints to them.

Would daemonizing be a better solution? If so I'd like some help understanding why.

Answer 1

There are a lot of good reasons to go with your solution:

• It's easier to think about than signals.
• It's got fewer cross-platform issues to deal with.
• You've already got code that works this way.
• It makes it easy to add a "graceful shutdown" mechanism if you want to in the future.

… and so on.

Keep in mind that, unless you can prove to yourself that multiprocessing and the underlying OS primitives, on every platform you care about, are guaranteed to work without synchronization here, you need to put a Lock or something else around every access to the shared bool. That isn't exactly complicated, but… once you've done that, using, eg, an Event without the shared bool might be even simpler.

At any rate, if any of those were your reason, I'd say great, do it that way. But according to your question, you actually chose this because of performance:

I feel like there is less overhead for all the children to just look at the one shared bool, than to send x number of sigints to them

If that's your reason, you're almost certainly wrong. The children have to look at the shared bool (and acquire the shared lock!) every time through some loop, while a signal only has to be sent to each child once. So, your overhead is almost certainly going to be much higher this way.

But really, I can't imagine the overhead of sending one signal per child process, or even grabbing an interprocess lock once per loop per process, is anywhere close to a bottleneck in any useful program, so… why does the overhead even matter here in the first place? Do what makes the most sense in the most simple way.

Answer 2

Since you are careful about who modifies the shared variable, it should be fine.

There are many different solutions possible. Eg use a multiprocessing.Event , and have the processes terminate when it is set. Or using multiprocessing.Connection objects (from Pipe). The latter could be used for two-way communication between parent and children. Like a signal to the children to stop, followed by a confirmation to the parent.

Answer 3

The people who tell you "don't do this" are wrong. The point of shared memory is to share memory among multiprocessors and that's exactly what you are doing.

You have a solution that 1) is simple, and 2) works. The signal/daemon approach is 1) really cool and 2) harder to code correctly and 3) much harder to understand.

The only pitfall I see in your approach is the possibility that a process could see a stale copy of the bool from the CPU's cache, and be delayed slightly in shutting down. There are ways to flush cache to ensure that this is not happening, but you probably don't need them because for most applications the cache flushing happens often enough automatically.

Stand your ground.