Long running tasks with Python and Flask

Question

I have a rather large database import job that I start from a web interface (Flask). When visiting the URL that trigger the database import, I fork before doing database inserts.

The problem is that when I attempt to stop the web service (Ctrl+C on the built in development server) the child process stops instead, and the web server keep running in the background. Somehow it appears as if the web server job is suddenly pushed to becoming a background daemon while the child process now is the primary process.

What I want to achieve is a "start child process and then forget about it" approach where the web server just starts the child process and then never bother about it again, especially when it comes to exceptions and similar.

Any ideas on how this is best solved?

Currently the test code for the import function is:

def import_start():
    try:
            pid = os.fork()
    except OSError as e:
            print "Exception in import_start"
            sys.exit(1)
    if pid == 0:
          with open("/tmp/web_out.txt", "w") as f:
                for x in range(100):
                      f.write("line %d\n" % (x))
                      f.flush()
                      sleep(10)

The code is started in a flask route handler with:

import_start()

Using Ctrl+C afterwards will kill the import_start() process instead of the web server. I want it to be the other way around as the two processes should be completely independent of each other after launch.

UPDATE:

I ended up doing:

def start_import():
            subprocess.Popen([sys.executable,__file__],stdout=subprocess.PIPE,stderr=subprocess.STDOUT)

def do_import():
    with open("/tmp/web_out.txt", "w") as f:
            for x in range(100):
                    f.write("line %d\n" % (x))
                    f.flush()
                    sleep(10)

if __name__ == "__main__":
    do_import()

This works (except for killing the spawned process when my web server is killed), but I am a bit worried about not cleaning up the process somehow. I might look into the MQ approach to solve this, although this requires more complexity than just spawning a new process.

Answer 1

Maybe python-daemon could help. This is mainly used to start a python program as daemon. Afaik it does the double fork magic and detaches the process from the current one.