Storing logging information from a Celery task in Django database

Question

Let's say I have the following function that I want to execute periodically in Celery:

import logging
import requests

log = logging.getLogger(__name__)


def spam():
    data = fetch_some_data()
    log.info(f'Fetched {len(data)} data')

    stuff = []
    for item in data:
        try:
            response = requests.get(url + item)
            response.raise_for_status()
        except RequestException as exc:
            log.error(f'Error when requesting {item}: {exc}')
            continue

        stuff.append(response.text)

    for item in stuff:
        do_something(item)
    log.info(f'Processed {len(stuff)} items')

When the accompanying task is executed:

from hello import spam

@app.task
def run():
    spam()

I would like to see the result of the task stored along with the logged messages. Flower for example is able to show the task progress and history but I would like to add the logging information as separate fields (eg "Info Messages" and "Error Messages"). Similarly, if using django-celery-results , it could show the same information in the Django Admin view.

What's the best way to achieve this?

Answer 1

Just take advantage of the after_setup_logger decorator. You can register your logger on celery startup so that you can use it inside your celery tasks.

@after_setup_logger.connect
def setup_loggers(logger, *args, **kwargs):
    import logging

    fh = logging.FileHandler('/path/to/file')
    logger = logging.getLogger("test-logger")
    logger.addHandler(fh)

Now, you can use the "test-logger" logger inside your tasks and redirect the logs into a file.