无法将Celery任务日志推送到Google Cloud（堆栈驱动程序）

Question

I'm trying to push celery task logs to GCP using their Logging API. I'm using google.cloud.logging.handlers.CloudLoggingHandler for the same.

Here is how I'm doing it:

settings.py

from google.cloud import logging as gcp_logging
gcp_client = gcp_logging.Client()
gcp_client.setup_logging()
LOGGING_FOLDER = ''
LOG_NAME = 'custom-logs'
LOGGING = {
    'version': 1,
    'disable_existing_loggers': True,
    'formatters': {
        'standard': {
            'format': '%(asctime)s [%(levelname)s] %(name)s: %(message)s'
        }
    },
    'handlers': {
        'gcp_log': {
            'level': 'DEBUG',
            'class': 'google.cloud.logging.handlers.CloudLoggingHandler',
            'client': gcp_client,
            'name': LOG_NAME,
            'formatter': 'standard',
        },
        'celery': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': os.path.join(LOGGING_FOLDER, 'celery.log'),
            'maxBytes': 1024 * 1024 * 100,  # 100 MB
            'backupCount': 15,
            'formatter': 'standard',
        }
    },
    'loggers': {
        'myapp.tasks':{
            'handlers': ['celery', 'gcp_log'],
            'level': 'DEBUG',
            'propagate': True
        },
        'celery.worker':{
            'handlers': ['celery', 'gcp_log'],
            'level': 'DEBUG',
            'propagate': False
        },
    },
}

myapp/tasks.py

from celery.utils.log import get_task_logger
logger = get_task_logger(__name__)
@task
def add(x, y):
    print("Add - I am from Print ")
    logger.info("Add - I am from logger")
    return x + y

All the logs, appear in celery handler ie celery.log file but only worker logs appear on GCP. myapp.tasks logs doesn't get uploaded to GCP.

My app is currently django based but I've also tried it in vanilla Python. Still the same behaviour.

I followed the method 1 described in https://www.distributedpython.com/2018/08/28/celery-logging/ but it also yielded same results. Is there any configuration miss I'm doing?

Answer 1

I encountered a similar problem a few weeks ago but with logging to splunk. Here is my answer: How can I log from my python application to splunk, if I use celery as my task scheduler?

I think that the problem could be where you set:

logger = get_task_logger(__name__)

This might generate a logger that is not mentioned in the settings.py file. Maybe grab a logger that you specificaly set up in your settings.py file with its handlers. But I personnaly just set up my own logger like you can see in my linked answer from above. I hope this might help yuo a bit...