如何重启Celery Wroker由Supervisord运行

Question

I am running celery on production using supervisord. My supervisor configuration is below.

[program:celeryd]
command=%(ENV_PROJECT_PATH)s/scripts/celery_worker.sh
stdout_logfile=%(ENV_PROJECT_PATH)s/celeryd.log
stderr_logfile=%(ENV_PROJECT_PATH)s/celeryd.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=1000
priority=1000

My command to run celery worker is

celery_path=$(which celery)
$celery_path -A Project_Name worker --loglevel=info

I want to ask, how to restart celery worker when my codebase changes in production?

Answer 1

Add following in supervisor file and restart supervisor.

killasgroup=true

Answer 2

The main issue I run into is that long running tasks may get killed if you tell supervisor to killasgroup which would result in lost data.

The solution I've moved to using is to tell the mainprocess to TERM which will kill off the workers as they finish their tasks. supervisor will then restart the main process after all the workers finish.

ps aux | grep celery.*MainProcess | awk '{print $2}' | xargs kill -TERM

This is also related. Celery Production Graceful Restart