How to shut down a Python TCPServer or HTTPServer or SimpleHTTPServer?

Question

I'm putting together a test harness for a C/C++ HTTP/HTTPS client, and I'd like to use python for the server in the test harness. Python seems to have a very nice set of TCP/HTTP server and handler classes and seems like it would be the path of least resistance for getting this done.

That said, I'm trying to figure out how to gracefully shut down a TCPServer object. My test harness is written in C++ and I'd like to use popen() to invoke the server, just because it isn't as problematic as system() and its available on all my target platforms (Windows, OSX, Linux). Given that I'm using popen() to launch the python server, I won't have the python process's PID available to me, so I can't easily SIGTERM the server.

I thought that I'd like to send a 'q' character to the server process's STDIN, or shut the STDIN pipe to the server sending it EOF to get it to gracefully shut down. I just can't figure out a really great way of doing this.

TCPServer / HTTPServer /et al start their work when you call their .serve_forever() method. So I guess I shouldn't be surprised that it's difficult to get TCPServer to stop serving forever when I call a method on it called .serve_forever() . As an alternative, the python docs for BaseHTTPServer suggest repeatedly calling .handle_request() instead:

def run_while_true(server_class=BaseHTTPServer.HTTPServer,
                   handler_class=BaseHTTPServer.BaseHTTPRequestHandler):
    """
    This assumes that keep_running() is a function of no arguments which
    is tested initially and after each request.  If its return value
    is true, the server continues.
    """
    server_address = ('', 8000)
    httpd = server_class(server_address, handler_class)
    while keep_running():
        httpd.handle_request()

But that's doesn't work very well either - it relies on the server receiving and finishing a request to check whether to keep running - if the server doesn't get a request, it doesn't return from .handle_request() .

TCPServer does have a method called .shutdown() :

BaseServer.shutdown()
    Tell the serve_forever() loop to stop and wait until it does.

    New in version 2.6.

Is this thread-safe? I know about the GIL, but I'd still like to know whether this is considered The Right Thing To Do , in light of alternative python implementations like PyPy. Could I wait on STDIN on another thread, and then call the TCPServer 's .shutdown() method safely when I get EOF on STDIN?

Answer 1

Yes, the BaseServer.shutdown() method is thread-safe.

The serve_forever method serves until the shutdown flag is set, and it is shutdown() s task to set that flag. The method states, in the docstring:

Blocks until the loop has finished. This must be called while serve_forever() is running in another thread, or it will deadlock.

Note that this has nothing to do with the GIL; that only limits how many cores a Python process can effectively make use of outside of C extensions and I/O.