Celery Tasks with eta get removed from RabbitMQ

Question

I'm using Django 1.6 , RabbitMQ 3.5.6 , celery 3.1.19 .

There is a periodic task which runs every 30 seconds and creates 200 tasks with given eta parameter. After I run the celery worker, slowly the queue gets created in RabbitMQ and I see around 1200 scheduled tasks waiting to be fired. Then, I restart the celery worker and all of the waiting 1200 scheduled tasks get removed from RabbitMQ .

How I create tasks: my_task.apply_async((arg1, arg2), eta=my_object.time_in_future)

I run the worker like this: python manage.py celery worker -Q my_tasks_1 -A my_app -l

CELERY_ACKS_LATE is set to True in Django settings. I couldn't find any possible reason.

Should I run the worker with a different configuration/flag/parameter? Any idea?

Answer 1

As far as I know Celery does not rely on RabbitMQ's scheduled queues. It implements ETA/Countdown internally.
It seems that you have enough workers that are able to fetch enough messages and schedule them internally.
Mind that you don't need 200 workers. You have the prefetch multiplier set to the default value so you need less.