How to use a Python script to start and stop a Python script containing “http.server.serveforever”

Question

Background

I have a Python 3 script called server.py that uses the built-in http.server module. The script boils down to the following:

from http.server import HTTPServer
from http.server import BaseHTTPRequestHandler

class MyRequestHandler (BaseHTTPRequestHandler):
    def do_POST(self):
        # Code omitted for brevity

    def do_GET(self):
        # Code omitted for brevity

def start_server():

    # Begin serving
    # -------------
    server = HTTPServer(('', port), MyRequestHandler)
    print("server now running on port {0} ...".format(port))

    server.serve_forever()

# Start the Server
# ----------------
if __name__ == '__main__':
    start_server()

MyRequestHandler handles GET and POST requests by importing modules "on-the-fly" depending on the URI that was used for the request.

The above works fine, however, subsequent to the creation of that script, there has been a requirement to be able to remotely update the whole "package" of scripts (ie the server script, plus all of the "module scripts" that are loaded up "on-the-fly" and are located in sub-folders).

For this, I have written another server script in Python 3 (called updater.py ) which, when instructed, will retrieve a zip file, which it then unzips to overwrite the original server.py script along with all the other associated scripts and sub-folders.

Question

This all works great, but I have now hit a wall. The best approach to this, I thought, would be to have the updater.py script control the running of server.py . It could shut down the server.py , and everything linked to it before overwriting it all and then give it a clean start after it is overwritten.

On that basis, the road that I have gone down is to use subprocess.Popen to start the server, believing that I could just kill the Python process before overwriting the server.py stuff, however, this is not working as hoped. Here is my trial.py script that I have written to test the theory:

import sys
import subprocess

def main():
    def start_process():
        proc = subprocess.Popen([sys.executable, 'server.py'])
        print("Started process:")
        print(proc.pid)
        return proc

    def kill_process(the_process):
        print("Killing process:")
        print(the_process.pid)
        the_process.kill()

    process = None

    while True:
        user_input = input("Type something: ")

        if user_input == 'start':
            process = start_process()
        if user_input == 'kill':
            kill_process(process)
        if user_input == 'exit':
            break

if __name__ == '__main__':
    main()

This does appear to start and kill a Python process, but the server is not running while this script is running, so I am not sure what it is starting and killing! Typing "start" and then "exit" (and thus quitting the trial.py script) allows the server to run, though I can't understand why, since I thought subprocess.Popen should cause the spawned process to run independently of the parent process?

edit: Thanks to @Håken Lid's astute observation below, I noticed that all I was doing is breaking out of the while loop, not exiting the script. This leads me to believe that the while loop is somehow blocking the sub-process from running (since once the loop is exited, the server will start).

Answer 1

per our discussion, I'd recommend some way to empty the stdio buffers from "server.py". If you want to also be able to give user input, you'll need a thread to do the printing (or just to empty the buffers into a black hole) while you wait for user input on the main thread. Here's a rough idea of how I might do it..

import sys
import subprocess
from threading import Thread 
#This could probably be solved with async, but I still
#haven't learned async as well as I know threads

def main():

    def start_process():
        proc = subprocess.Popen([sys.executable, 'server.py'], 
                                stdin=subprocess.PIPE, 
                                stdout=subprocess.PIPE)
        print("Started process:")

        def buf_readerd(proc, inbuf, outbuf):
            while proc.poll() is None:
                outbuf.write(inbuf.readline()) #may need to add a newline.. I'm not sure if readline ends in a \n

        stdoutd = Thread(target=buf_readerd, args=(proc, proc.stdout, sys.stdout), daemon=True)
        stderrd = Thread(target=buf_readerd, args=(proc, proc.stderr, sys.stderr), daemon=True)
        stdoutd.start()
        stderrd.start()
        print("started pipe reader daemons")

        print(proc.pid)
        return proc
# ...

Answer 2

Okay, so I think I have fixed this myself now. The problem was that I was blocking the sub-process execution with the call to input() inside the while loop. The script does nothing until it receives some input. I made subprocess.poll() an integral part of the loop and put the call to input() inside this loop. If I hit <return> a few times, the server will start and I can use it. If I then type kill , the server will be killed, as expected.

I am not sure how this will work in the context of my updater.py script, but I have learnt a valuable lesson in "thinking more deeply before posting to StackOverflow"!

import sys
import subprocess

def main():
    def start_process():
        proc = subprocess.Popen([sys.executable, 'server.py'])
        print("Started process:")
        print(proc.pid)
        return proc

    def kill_process(the_process):
        print("Killing process:")
        print(the_process.pid)
        the_process.kill()

    user_input = input("Type something: ")

    if user_input == 'start':
        process = start_process()

        while process.poll() is None:
            user_input = input()
            if user_input == 'kill' or user_input == 'exit':
                kill_process(process)

if __name__ == '__main__':
    main()