Clearing out redis from celery tasks that has finished

Question

I'm using celery task queue which use Redis as a message broker to run some background tasks that save results to Redis.

I wanted to clear out Redis from task results that did finish because i no longer need them and so in case of having a big spike I shouldn't worry.

I came across CELERY_RESULT_EXPIRES which indicates that a task result will automatically clean up after X seconds.

Consider the following code:

from celery import Celery

CONFIG = {
    'BROKER_URL': 'redis://localhost:6379/0',
    'CELERY_RESULT_BACKEND': 'redis://localhost:6379/0',
    'CELERY_RESULT_EXPIRES': 15, # 15 secs
    'BROKER_POOL_LIMIT': 0, # redis connection get closed once task is done..
}


celery = Celery('tasks', config_source=CONFIG)

@celery.task(name='tasks.say_hello')
def say_hello():
    return "Helloooooo i just came out from the background!"


if __name__ == '__main__':
    say_hello.delay().get()

When running the code above and having celery beat -A tasks -l info running too, then checking Redis after a while (more than 5mins) I see this:

127.0.0.1:6379> KEYS *
1) "_kombu.binding.celery"
2) "_kombu.binding.celery.pidbox"
3) "_kombu.binding.celeryev"
4) "celery-task-meta-854543d8-14ad-4bf8-9725-edcf64131bb2"
5) "celery-task-meta-fa3e267e-46d0-4488-a766-d3276b6abdeb"
6) "celery-task-meta-86c2d83c-cadd-41b9-b4ff-426607786299"

Tasks 4 to 6 which were completed still there, Isn't the results supposed to clear out after 15 seconds? Am I missing something here?

Thanks in advance.

Answer 1

@zhong So the thing you are missing is celery.backend_cleanup as stated in docs

"A built-in periodic task will delete the results after this time", so built in tasks run by default at 4 am daily, so if you don't change the configuration it will be deleted for sure, you can cross check that after 4am, in case if you are planning to store that in db you have to run celery beat for that,

So to recap it will be deleted yes, but only when celery backend cleaner will run, the one which you are configuring determines lifetime of that result that need to be stored, but that doesn't mean redis will delete that for you, that expiry team is for celery cleaner to look and determine that, yes this result now can be deleted.