How to reset time limit on Celery Task

Question

I have no experience with Celery so I'm looking into it if my use case is solvable in Celery.

The client will submit a job to Celery, this job will be executed in CeleryTask. Then the client has to send keepalive every 30 seconds to keep this job active. Once the job is not refreshed by keepalive message, the job will be cancelled.

I can think of two solutions:

1. Each job-task created will have hard time limit of 30s. When client sends keepalive the router will send message to relevant worker to reset the hard time limit.

2. Each job-task will have no time limit. For each job-task there will be another special watchdog task launched. The watchdog task will be launched with delay of 30s. If new keepalive arrives from the client the watchdog task is cancelled and recreated. Again with delay of 30 seconds. If the watchdog is executed, it will kill the job-task, thus eliminating it from system.

The 1. is much simpler, but I'm not sure how to reset the task timelimit. The solution 2. seems more correct, but I'm afraid there will be various race conditions. The watchdog task should be probably running in separate queue reserved for watchdogs only.

How is something like this possible? Even one of my solution or some other.

Answer 1

In my understanding, you just want to have a middleware (one to receive keepalive , on to control task), then this link will help you celery.app.control.Control.terminate .

1. Get task_id of task when apply_async ( destination or all workers )
2. listen the keepalive from client
3. if time limit arrived and no keepalive , terminate task app.control.terminate(task_id, reply=True)