Why is Celery not shutting down cleanly?

Question

After issuing TERM signal to the Celery worker twice (Warm shutdown and cold shutdown) using Ctlr-C keyboard interrupt, the Celery worker is just hung up. It's not consuming messages or executing tasks (as expected) but it's not shutting down either.

I ran strace on the Celery processes to see what's going on behind the scene. Here's the strace output on the PID of the Celery main process

strace -p 27867
Process 27867 attached - interrupt to quit
futex(0xb966a78, FUTEX_WAIT, 0, NULL

And here's what I found doing strace on the child processes:

strace -p 27874
Process 27874 attached - interrupt to quit
select(4, [3], NULL, NULL, {0, 562000}) = 0 (Timeout)
futex(0x871a808, FUTEX_WAKE, 1)         = 0
select(4, [3], NULL, NULL, {1, 0})      = 0 (Timeout)
futex(0x871a808, FUTEX_WAKE, 1)         = 0
......................................................

I know I could issue a KILL signal to the processes to get rid of them. But I'm curious to learn what is actually preventing these processes to be shutdown, and if it's possible to do something about it.

Software stack: Python 2.6.2, Celery 2.4.6, CentOS 5.0

UPDATE: The CPU usage is down to almost 0%. These tasks are quite CPU intensive, so this confirms no tasks are currently active.

Answer 1

From the docs :

If the worker won't shutdown after considerate time, for example because of tasks stuck in an infinite-loop, you can use the KILL signal to force terminate the worker, but be aware that currently executing tasks will be lost (unless the tasks have the acks_late option set).

Also from google groups :

celeryd will not shut down until all active tasks have been processed, where active means the tasks that have been started on (not all reserved tasks). The reserved messages will be released and redelivered once the connection channel is closed. This happens after the active tasks have returned. if you don't have a --time-limit enabled celeryd will never kill your tasks at shutdown, even if they take DAYS to complete.