Given a task_id, execute the task

Question

I'm creating a celery task in a situation where task producers are more than consumers (workers). Now since my queues are getting filled up and the workers consume in FCFS manner, can I get to execute a specific task(given a task_id ) instantly?

for eg:

My tasks are filled in the following fashion. [1,2,3,4,5,6,7,8,9,0] . Now the tasks are fetched from the zeroth index. Now a situation arise where I want to execute task 8 above all. How can I do this?

The worker need not execute that task (because there can be situation where a worker is already occupied). It can be run directly from the application. And when the task is completed (either from the worker or directly from the application), it should get deleted from the queue.

I know how to forcefully revoke a task (given a task_id ) but how can I execute a task given an id ?

Answer 1

how can I execute a task given an id ?

the short answer is you can't. Celery workers pull tasks off the broker backend as they become available.

Why not?

Note that's not a limitation of Celery as such, rather it is a characteristic of message queuing systems(MQS) in general. The point of MQS is to desynchronize an application's component so that the producer can go on to do other work while workers execute the tasks asynchronously. In other words, once a task has been sent off it cannot be modified (but it can be removed as long as it has not been started yet).

What options are there?

Celery offers you several options to deal with lower vs higher priority or short- and long-running tasks, at task submission time :

1. Routing - tasks can be routed to different workers. So if your tasks [0 .. 9] are all long-running, except for task 8, you could route task 8 to a worker, or a set of workers, that deal with short-running tasks.

2. Timed execution - specify a countdown or estimated time of arrival (eta) for each task. That's a good option if you know that some tasks can be delayed for later execution ie when the system will be less busy. This leaves workers ready for those tasks that need to be executed immediately.

3. Task expiry - specify an expire countdown or time with a callback. This way the task will be revoked if it didn't execute within the time alloted to it and the callback can start an alternative course of action.

4. Check on task results periodically , revoke a task if it didn't start executing within some time. Note this is different from task expiry where the revoking only happens once a worker has fetched the task from the queue - if the queue is full the revoking may happen too late for your use case. Checking results periodically means you have another component in your system that does this and determines an alternate course of action.