Scheduling celery tasks with large ETA

Question

I am currently experimenting with future tasks in celery using the ETA feature and a redis broker. One of the known issues with using a redis broker has to do with the visibility timeout :

If a task isn't acknowledged within the Visibility Timeout the task will be redelivered to another worker and executed.

This causes problems with ETA/countdown/retry tasks where the time to execute exceeds the visibility timeout; in fact if that happens it will be executed again, and again in a loop.

Some tasks that I can envision will have an ETA on the timescale of weeks/months. Setting the visibility timeout large enough to encompass these tasks is probably unwise.

Are there any paths forward for processing these tasks with a redis broker? I am aware of this question . Is changing brokers the only option?

Answer 1

I am doing this with redis in the following way:

We have customers that can schedule a release of some of their content. We store the release in our database with the time it should be executed at.

Then we use celery beat to perform a periodic task (hourly or what suits you) that checks our releases table for releases that are scheduled within the next period (again hour or what suits you). if any are found we then schedule a task for them with celery. This allows us to have a short ETA.