Limit on Python Celery eta?

Question

Does Celery have a limit on the eta for a task? I want to execute the method foo after 12 days, will Celery have a problem with that? Or do I need configure any Celery settings for such a long eta?

next_run = datetime.now() + timedelta(days = 12)
foo.apply_async(args=[], eta = next_run)

Answer 1

It depends on what broker you're using and a few configurations like visibility_timeout .

When a task with ETA set is enqueued, it is fetched by a worker asap, instead of at ETA. The worker holds the task until ETA and executes it then. The worker acks the broker when it handles the task; so the broker knows when the task message can be safely deleted.

Some brokers, like Redis and SQS, consider a task lost if it is not acknowledged by a worker for some time, which is called visibility_timeout . In such a case, it allows another worker to take the task message.

Back to your case, if you're using Redis, which has a default visibility_timeout of 1 hour, delaying a task 12 days may end up executing the task multiple times, like 12 * 24 times, depending on your number of workers.

It may be tempting to increase visibility_timeout to more than 12 days. Be aware that it effectively disables the acknowledgment mechanism, which exists for a good reason.

I suggest you use a cron task, which runs at an interval of visibility_timeout , to scan for tasks that should soon be executed and enqueue them at their actual ETA.

Refs:

• visibility_timeout
• acknowledement