django: Celery how to get a list of task results by their ids

Question

Wanted to check if there is a way to optimize the process of celery task results checking.

For example, I have the following code:

for task_id in tasks: # tasks is a list of celery task ids
    res = app.AsyncResult(task_id)
    if res.successful():
        tasks_ids[task_id] = res.result # here I am storing task results

The problem with current approach is that I have to loop over celery tasks and check them one by one. And as well as I am using a database results backend it will produce a query per each celery taskid (at least I think so).

So I wanted to check if there is a way to retrieve all results at once? Is there. Is there app.AsyncResult() analogue which operates with a list?

Answer 1

Celery has a ResultSet class to work with this, it has "join()" and "native_join()" methods to do that. There is a drawback as the doc says, the database backends does not implement the "native_join()" (these backends do implement it: amqp, Redis and cache), because of that the ResultSet could be a really expensive, similar to your current approach. By the way, I think you could take a look to the doc and API and see if you find something which could help you. I do not know how hard could be add that support to database backend (you would need to implement "get_many()" as the API says says. However, you could evaluate changing the backend.

I hope it helps you!