best way to run a script that takes a long time using an API call in Python

Question

I have an API that currently runs a script and returns a status code based on the output of the script. It also serves a file via another API upon completion of the script. However, in production I have found that the process could take a long time and the request could time out before the script finishes running. How can I make this API such that it wont wait for the script to finish and the script will send the file well after the first API call responded?

Answer 1

Web servers like to keep request/response cycles short. When some requests take increasingly longer to satisfy, there comes a point where that work needs to be "out of process" to avoid gumming up the web server.

This need is why things like celery exist. The idea is that a client describes the work to be done in a request. The handler will queue up that description, getting an id in return, and will pass that id back to the client in the response. The client remembers that id, and then polls, passing that id. Meanwhile, a worker process running elsewhere has picked up the work description from the queue, done done the work, and return the result. Eventually a poll request will say "done," instead of "please wait". and the client can retrieve the results.

The Flask Mega Tutorial , in chapter 22, covers a way way to do this using Rq , which is simpler, but less robust, than celery. It's worth looking at that chapter to solidify the general idea.