设置工作队列，使工作人员能够被杀死并重新排队

Question

I am looking for a way to setup a job queueing system in python, like RQ or Celery. I am using elastic beanstalk for deployment.

I am currently using RQ, but i am facing the following problem: if the worker gets killed, the job is lost. I would like it to be automatically requeued. I have long running jobs (some can last for 1h). Sometimes, we may need to reboot the server, or deploy a new version, without waiting for all jobs to finish. Sometimes, some essential container fails and cause all the other containers to be rebooted.

During reboot, Beanstalk automatically send the SIGINT signal to stop container, then after 10 seconds, send the SIGKILL signal. I cannot find a way to have the jobs requeued in such an event. I tried both RQ with Redis and Celery with a Redis Broker.

Can someone recommand a setup for this?

Answer 1

One thing you can do with Celery is to enable the task-reject-on-worker-lost configuration setting.

Even if task_acks_late is enabled, the worker will acknowledge tasks when the worker process executing them abruptly exits or is signaled (eg, KILL/INT, etc).

Setting this to true allows the message to be re-queued instead, so that the task will execute again by the same worker, or another worker.