Return HTTP response and continue processing - python bottle API

Question

I have an analysis service using Python's bottle that is invoked by posting a url to the raw data.

What is the simplest/cleanest/best way to immediately return a 200 response to the callee and continue processing a potentially long-running logic in the application?

Spawn a new thread/process? Use a queue? Async?

from bottle import post, HTTPResponse, request

@post('/postprocess')
def records():
   data_url = request.json.get('data_url', False)
   postback_url = request.json.get('postback_url', False)

   if not data_url or not postback_url:
       return HTTPResponse(status=400, body="Missing data paramater")

   #Immediately return response to callee
   return HTTPResponse(status=200, body="Complete")

   #Continue processing long-running code
   result = get_and_process_data(data_url)
   #POST result to another endpoint 

Answer 1

There is no simplest solution to this. For a production system I would first look into an existing system created for these kinds of situations to determine if that would be a good fit, and if not, only then develop something more suitable to my situation. To that end, I would recommend you take a look at Celery

Answer 2

I would suggest using some Queuing mechanism to queue the data that needs processing. Then you can implement a pool of workers that can work on processing the Q.

You can then take advantage of Q monitoring tools to iron out any performance issues and your worker pool can scale up as needed.

Answer 3

I've recently found a very straightforward solution to this.

from threading import Thread    
@app.route('/')  
def send_response():
    uri = request.args
    ua = request.headers['user-agent']
    token = request.args.get("token")
    my_thread = Thread(target=process_response, args=[uri,
                                                      ua,
                                                      token])
    my_thread.start()
    res = Response(status=200,
                   mimetype='application/json')
    return res
        
def process_response(uri, ua, token):
    print(uri)
    print(ua)
    print(token)
    pass