DJCelery not storing task results in Django SQLite DB

Question

DJCelery is not storing task results in my Django SQLite DB.

I have an existing Django project that I have started setting up Celery w/ RabbitMQ on. I started my RabbitMQ server. I can run Celery python manage.py celeryd --verbosity=2 --loglevel=DEBUG and Celerybeat python manage.py celerybeat --verbosity=2 --loglevel=DEBUG . Everything starts up w/ out error and my periodic example tasks also runs without error.

I used pip install django-celery to install. I have djcelery in my installed apps and ran python manage.py migrate djcelery . I added CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend' to the end of my settings.py file.

When I run python manage.py celeryd --verbosity=2 --loglevel=DEBUG , the startup text shows:

...
- ** ---------- .> transport: amqp://guest:**@localhost:5672//
- ** ---------- .> results:
- *** --- * --- .> concurrency: 1 (prefork)
...

The results section being blank indicates to me that the configuration isn't right somehow but I can't figure out how. I tried using app.conf.update in my celery.py file to set the CELERY_RESULT_BACKEND but got the same results. I left out CELERY_RESULT_BACKEND, but that defaulted to no results. I also tried putting 'database' instead of 'djcelery.backends.database:DatabaseBackend' but that indicated it was attempting to use sqlalchemy instead of djcelery .

When I run python manage.py runserver I can see a DJCELERY section with tables Crontabs , Intervals , Periodic tasks , Tasks , and Workers . There isn't any data on my Tasks though.

Can anyone point out what could be wrong or missing? Thank you for your time.

Answer 1

tutuDajuju led me in the right direction - there's more to it so I'll write it all up. I abandoned using djcelery in favor of sqlalchemy with a separate back-end database outside of Django .

Inside my venv I ran pip install sqlalchemy . I then put CELERY_RESULT_BACKEND = 'db+sqlite:///celery_results.sqlite3' in settings.py . This connected Celery to the new SQLite database to use for state/results.

Running celery -A <projectapp>.celery:app worker then showed the database in the startup message:

...
- ** ---------- .> transport: amqp://guest:**@localhost:5672//
- ** ---------- .> results: sqlite:///celery_results.sqlite3
- *** --- * --- .> concurrency: 1 (prefork)
...

At first I was worried because the database file wasn't created in my Django project dir. This is because I hadn't ran a task yet. Once I ran my first task, the database & tables were created correctly.

I verified task results were stored in the database by running a script:

from sqlalchemy import create_engine

engine = create_engine("sqlite:///celery_results.sqlite3")
connection = engine.connect()

result = connection.execute("select * from celery_taskmeta")

for row in result:
    print(row)

connection.close()

I found the table names by:

print(engine.table_name())

Hope this helps someone out.

Answer 2

The celery docs mention a few different syntaxes, not sure what you tried is valid. Try the following:

# use a connection string
CELERY_RESULT_BACKEND = 'db+sqlite:///foo.db'

Update :

As in your comment, the docs also mention it is also possible to use the Django ORM/Cache as a result backend. To do this, you must pass the setting you tried into your celery app config:

app.conf.update(
    CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend',
)

Alternatively, the docs also explain

If you have connected Celery to your Django settings then you can add this directly into your settings module (without the app.conf.update part)

This is a reference to the configuration of the Celery app detailed in the same page. This basically means that if you configured your celery app in a module, and you add the Django settings module as a configuration source for Celery, then setting CELERY_RESULT_BACKEND in your Django settings module, as you did, will also work.

file : proj/proj/celery.py

# important to pass the Django settings to your celery app
app = Celery('proj')
app.config_from_object('django.conf:settings')

file : proj/proj/settings.py

CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend'