Handling child process shutdown gracefully

Question

I am working on a project where I have a pool of workers. I am not using the built-in multiprocessing.Pool , but have created my own process pool.

The way it works is that I have created two instances of multiprocessing.Queue - one for sending work tasks to the workers and another to receive the results back.

Each worker just sits in a permanently running loop like this:

while True:
    try:
        request = self.request_queue.get(True, 5)
    except Queue.Empty:
        continue
    else:
        result = request.callable(*request.args, **request.kwargs)
        self.results_queue.put((request, result))

There is also some error-handling code, but I have left it out for brewity. Each worker process has daemon set to 1 .

I wish to properly shutdown the main process and all child worker processes. My experiences so far (doing Ctrl+C):

• With no special implementations, each child process stops/crashes with a KeyboardInterrupt traceback, but the main process does not exist and have to be killed ( sudo kill -9 ).
• If I implement a signal handler for the child processes, set to ignore SIGINT's, the main thread shows the KeyboardInterrupt tracebok but nothing happens either way.
• If I implement a signal handler for the child processes and the main process, I can see that the signal handler is called in the main process, but calling sys.exit() does not seem to have any effect.

I am looking for a "best practice" way of handling this. I also read somewhere that shutting down processes that were interacting with Queue s and Pipe s might cause them to deadlock with other processes (due to the Semaphores and other stuff used internally).

My current approach would be the following: - Find a way to send an internal signal to each process (using a seperate command queue or similar) that will terminate their main loop. - Implement a signal handler for the main loop that sends the shutdown command. The child processes will have a child handler that sets them to ignore the signal.

Is this the right approach?

Answer 1

The thing you need to watch out for is to deal with the possibility that there are messages in the queues at the time that you want to shutdown so you need a way for your processes to drain their input queues cleanly. Assuming that your main process is the one that will recognize that it is time to shutdown, you could do this.

1. Send a sentinel to each worker process. This is a special message (frequently None ) that can never look like a normal message. After the sentinel, flush and close the queue to each worker process.

2. In your worker processes use code similar to the following pseudocode:

    while True: # Your main processing loop msg = inqueue.dequeue() # A blocking wait if msg is None: break do_something() outqueue.flush() outqueue.close() 

If it is possible that several processes could be sending messages on the inqueue you will need a more sophisticated approach. This sample taken from the source code for the monitor method in logging.handlers.QueueListener in Python 3.2 or later shows one possibility.

            """
            Monitor the queue for records, and ask the handler
            to deal with them.

            This method runs on a separate, internal thread.
            The thread will terminate if it sees a sentinel object in the queue.
            """
            q = self.queue
            has_task_done = hasattr(q, 'task_done')
            # self._stop is a multiprocessing.Event object that has been set by the
            # main process as part of the shutdown processing, before sending
            # the sentinel           
            while not self._stop.isSet():
                try:
                    record = self.dequeue(True)
                    if record is self._sentinel:
                        break
                    self.handle(record)
                    if has_task_done:
                        q.task_done()
                except queue.Empty:
                    pass
            # There might still be records in the queue.
            while True:
                try:
                    record = self.dequeue(False)
                    if record is self._sentinel:
                        break
                    self.handle(record)
                    if has_task_done:
                        q.task_done()
                except queue.Empty:
                    break