Celery - How to update the state of a task after the worker's being shutdown?

Question

Setup

Celery 3.0

broker=RabbitMQ

Scenario

Tasks have already been acknowledged and started processing, and have state=STARTED . Then I want to restart my worker (to update the worker to a newer version). After restart worker (using supervisorctl restart ), those long running tasks are all been terminated. But their states remain in state=STARTED . How can I update their state to FAILURE or whatever other values? (And, I don't want these tasks being executed again after the worker restarts.)

Methods tried (but not working)

• use track_started=True --- If with this option, the tasks stay at state=STARTED after the worker restarts. If without this option, the tasks stay at state=PENDING after the worker restarts.
• use CELERY_ACKS_LATE=True --- The tasks stay at state=STARTED after the worker restarts. And tasks are executed again, not a desired behavior.
• use signal(SIGTERM, handler) and a handler function to catch the signal. The handler can successfully be entered. However, no matter what thing I put inside the handler, it can't change the task's state. The states just stay as the same and won't change to FAILURE . Inside the handler I've tried
  • raise Exception
  • exit(0)
  • exit(1)

Is there any settings of Celery that could enable it to track the state of task being shutdown?

Answer 1

You need to revoke the tasks on worker shutdown. Take a look at this issue for the actual code.

Answer 2

I believe the better way would be adjust stopwaitsecs ( see ) in your supervisor config to be more than your time limit per task. Supervisor would wait for your tasks to finish without killing them. So your tasks always finished normally and no states to fix in the first place.

Also this depends on how long your tasks are. And if they too long running to wait for them, may be it's better to split them to shorter ones.

Answer 3

I find the last method ( signal(SIGTERM, handler) ) you tried working. Since the code successfully enter the TERM signal handling branch, the crux becomes how to update the state of task. Simply use self.update_state(state=states.FAILURE) inside the task body.

Here's an example:

import time    
import signal
from celery import Celery, states

app = Celery("celery")

@app.task(bind=True)
def demo(self):
    
    def mark_as_fail(*args):
        self.update_state(state=states.FAILURE)

    signal.signal(signal.SIGTERM, mark_as_fail)

    for i in range(30):
        print(f"I am demo task {i}")
        time.sleep(1)