Store references to Dask Futures in a Flask/FastAPI Application

Question

I'm building a FastAPI application that has an endpoint to trigger a Dask computation. The API endpoint sends this call to the Dask scheduler and just returns the key of the Future .

trigger

x = client.submit
    (
        function_name,
        arg1,
        arg2
    )
return x.key

I have two other endpoints to retrieve the status and result of the task, which take the key as input.

status

status = Future(key=key, client=client).status
return status

result

result = Future(key=key, client=client).result()
return result

Of course, this way, I lose the reference to the future after trigger returns, in which case Dask doesn't compute this anymore. So even if the key is given to the client, we get the status as pending forever.

What I'm doing now is storing references to the Future object as a python dictionary in the application, which works. But ideally, I would like my API application to be stateless. What would be store these Futures outside this application? Are there good caching libraries in Python that can store Python objects (with references)?

Answer 1

Try using Flask + Celery to handle the background Dask computation. Below are few links for reference:

1. https://flask.palletsprojects.com/en/1.1.x/patterns/celery/
2. https://blog.miguelgrinberg.com/post/using-celery-with-flask
3. https://medium.com/@frassetto.stefano/flask-celery-howto-d106958a15fe