TProcessPoolServer 正常关闭？

Question

How do you gracefully shutdown the python thrift server, TProcessPoolServer? I haven't found any documentation, examples, or blog posts. What follows are my experiences, so far.

I'm running my thrift server directly on the command line, ./thrift_service.py, not under a supervisor. I'm using python 2.6 and thrift 0.8.0.

I initially tried:

server = TProcessPoolServer(processor, transport, tfactory, pfactory)
try:
    server.serve()
finally:
    server.stop()

When I send sigterm the parent python process, I see "Terminated" in the output, the process is killed, but its children are orphaned and continue to run.

Then I stumbled across the thrift server tests , and tried:

import signal
def set_alarm(server):
    def clean_shutdown(signum, frame):
        for worker in server.workers:
            logging.error("Terminating worker: {0}".format(worker))
            worker.terminate()
        logging.error("Requesting server to stop()")
        try:
            server.stop()
        except (KeyboardInterrupt, SystemExit):
            pass
        except Exception as err:
            logging.exception(err)
    def logme(s, *args, **kwargs):
        logging.error(">>> {0} <<<".format(s))
        clean_shutdown(*args, **kwargs)
    signal.signal(signal.SIGALRM, clean_shutdown)
    signal.signal(signal.SIGHUP, clean_shutdown)
    signal.signal(signal.SIGINT, clean_shutdown)
    signal.signal(signal.SIGTERM, lambda x, y: logme("SIGTERM", x, y))
server = TProcessPoolServer(processor, transport, tfactory, pfactory)
set_alarm(server)
server.serve()

and when I send sigterm, sigalrm, sighup, or sigint to the parent python process, the server stops accepting connections, but the processes are not terminated.

In the output I see:

ERROR:root:>>> SIGTERM <<<
ERROR:root:Terminating worker: <Process(Process-1, started daemon)>
ERROR:root:Terminating worker: <Process(Process-2, started daemon)>
ERROR:root:Terminating worker: <Process(Process-3, started daemon)>
ERROR:root:Terminating worker: <Process(Process-4, started daemon)>
ERROR:root:Terminating worker: <Process(Process-5, started daemon)>
ERROR:root:Requesting server to stop()

which is expected, but then the signal is caught again, the processes aren't in a started state anymore, and the server is asked to stop. This part happens around ten times and then there is no more output.

ERROR:root:>>> SIGTERM <<<
ERROR:root:Terminating worker: <Process(Process-1, unknown daemon)>
ERROR:root:Requesting server to stop()

And sometimes, I see an AssertionError from within the multiprocessing library:

Traceback (most recent call last):
  File "/path/to/thrift_service.py", line 340, in clean_shutdown
    server.stop()
  File "/usr/local/lib/python2.6/dist-packages/thrift/server/TProcessPoolServer.py", line 123, in stop
    self.stopCondition.notify()
  File "/usr/lib/python2.6/multiprocessing/synchronize.py", line 223, in notify
    assert not self._wait_semaphore.acquire(False)
AssertionError

Answer 1

I have added a graceful shutdown to a TProcessPoolServer in python using signals and the postForkCallback that it exposes. The TProcessPoolServer will call your postForkCallback in each worker process once it has initialized. This allows you to setup signal handlers and shutdown gracefully. Since the workers catches either the SystemExit or KeyboardInterruptException exceptions you can setup a handler for SIGINT and then once you have finished cleaning up call sys.exit(0) and that will cause the worker to shutdown.

import signal
import sys

def setupHandlers():
    signal.signal(signal.SIGINT, handleSIGINT)
    #Optionally if you want to keep the current socket connection open and working
    #tell python to make system calls non-interruptable, which is probably what you want.
    signal.siginterrupt(signal.SIGINT, False)

def handleSIGINT(sig, frame):
     #clean up state or what ever is necessary
     sys.exit(0)

server = TProcessPoolServer(processor, transport, tfactory, pfactory)
server.setPostForkCallback(setupHandlers)

#Setup handlers in main process too
setupHandlers()

#Start server
server.start()

This way every worker process that is spawned sets the signal handlers to correctly handle the graceful shutdown. In this example I set the same handler for the main process as well as the workers which may work depending on you use case, but you can easily define a different handler for the main process if you need to. And remember that the handler will be called from the context of each process so you won't be able to share state across process during clean up.

see http://docs.python.org/library/signal.html for more details on what signal.siginterrupt does and why you may need it.

Edit: You will need to send the SIGINT signal to all of the process using Crtl + C or if it is running as a daemon kill -SIGINT [pids of all processes]

You can get the pids of the workers easily using ps --ppid [parent pid]

Answer 2

After the program started, I recorded the process number of the main process. Then according to ps --ppid , get back the child processes of the main process and kill them one by one.

The code of the control shell script of my service:

function stop
{
    SERVER_PID=`cat logs/server.pid`
    SPIDS=`ps --ppid $SERVER_PID | awk '{if ($1!="PID") print $1}'`
    kill -9 $SERVER_PID
    for PID in $SPIDS
    do
        kill -9 $PID
    done
}