Can't get two Python loggers to work at the same time

Question

Related to a previous question I asked, Airbrake logger in Django still sending notifications even though its level is set to 'CRITICAL'? , I'm working on a Django project with multiple versions of settings.py : settings/base.py , settings/staging.py , etc.

There is also a kind of settings 'mixin', settings/staging_development.py , which contains the following LOGGING configuration:

# Auxiliary variable used in LOGGING
_AIRBRAKE_LOGGER = {
    'handlers': ['airbrake'],
    'level': 'ERROR',
    'propagate': True,
}

# Airbrake logging integration (cf. https://github.com/airbrake/pybrake#django-integration)
# In our case, 'app' is replaced by three apps, 'lucy_web', 'api', and 'activation'.
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'airbrake': {
            'level': 'ERROR',
            'class': 'pybrake.LoggingHandler',
        },
    },
    'loggers': {
        'lucy_web': _AIRBRAKE_LOGGER,
        'api': _AIRBRAKE_LOGGER,
        'activation': _AIRBRAKE_LOGGER,
    },
}

This is imported in settings/staging.py in which LOGGING is further .update() d as follows:

from lucy.settings.staging_production import *


# LOGGING = {
#     'version': 1,
#     'disable_existing_loggers': False,
#     'handlers': {
#         'console': {
#             'class': 'logging.StreamHandler',
#         },
#     },
#     'loggers': {
#         'django': {
#             'handlers': ['console'],
#             'level': os.getenv('LOG_LEVEL', 'INFO'),
#         },
#     },
# }

LOGGING['handlers'].update(console={
    'class': 'logging.StreamHandler'
})
LOGGING['loggers'].update(django={
    'handlers': ['console'],
    'level': os.getenv('LOG_LEVEL', 'INFO'),
})

What I'd like to achieve is to log to Airbrake as well as to the console. Now, if I simply comment in the commented-out code and re-define the LOGGING configuration, I notice that error messages are successfully getting logged to the console. If I use the code as-is, however, they do not, even though I do get Airbrake notifications.

If I poke around in the shell, everything looks OK: for example, the 'lucy_web' and 'django' loggers both have handlers attached and propagate set to True :

(venv) Kurts-MacBook-Pro-2:lucy-web kurtpeek$ ENV_ROLE=staging_on_localhost python manage.py shell
Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.3.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from django.conf import settings

In [2]: settings.LOGGING
Out[2]: 
{'version': 1,
 'disable_existing_loggers': False,
 'handlers': {'airbrake': {'level': 'ERROR',
   'class': 'pybrake.LoggingHandler'},
  'console': {'class': 'logging.StreamHandler'}},
 'loggers': {'lucy_web': {'handlers': ['airbrake'],
   'level': 'ERROR',
   'propagate': True},
  'api': {'handlers': ['airbrake'], 'level': 'ERROR', 'propagate': True},
  'activation': {'handlers': ['airbrake'],
   'level': 'ERROR',
   'propagate': True},
  'django': {'handlers': ['console'], 'level': 'ERROR'}}}

In [3]: import logging

In [4]: logger = logging.getLogger('lucy_web')

In [6]: logger.propagate
Out[6]: True

In [7]: logger.hasHandlers()
Out[7]: True

In [8]: logger.handlers
Out[8]: [<LoggingHandler (ERROR)>]

In [9]: django_logger = logging.getLogger('django')

In [10]: django_logger.handlers
Out[10]: [<StreamHandler <stderr> (NOTSET)>]

In [11]: django_logger.getEffectiveLevel()
Out[11]: 40

In [12]: logging.ERROR
Out[12]: 40

In [13]: django_logger.propagate
Out[13]: True

In short, I can get one type of logging to work or the other, but not both at the same time, even though log propagation is enabled.

Any idea what the issue might be?

Answer 1

I resolved the problem by adding the 'console' handler to the apps ( 'lucy_web' , 'api' , and 'activation' ) themselves, rather than to the catch-all 'django' logger. Actually, I think that at the root of the problem, the errors I was throwing were not in 'django' logger's hierarchy.

So firstly, in settings/base.py I use deepcopy :

from copy import deepcopy

# Auxiliary variable used in LOGGING
_AIRBRAKE_LOGGER = {
    'handlers': ['airbrake'],
    'level': 'ERROR',
    'propagate': True,
}

# Airbrake logging integration (cf. https://github.com/airbrake/pybrake#django-integration)
# In our case, 'app' is replaced by three apps, 'lucy_web', 'api', and 'activation'.
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'airbrake': {
            'level': 'ERROR',
            'class': 'pybrake.LoggingHandler',
        },
    },
    'loggers': {
        # The deepcopy allows us to append to each app's 'handlers' list without affecting the others
        'lucy_web': deepcopy(_AIRBRAKE_LOGGER),
        'api': deepcopy(_AIRBRAKE_LOGGER),
        'activation': deepcopy(_AIRBRAKE_LOGGER),
    },
}

And secondly, in settings/staging.py I updated LOGGING like so:

LOGGING['handlers'].update(console={
    'class': 'logging.StreamHandler'
})

for app in ('lucy_web', 'api', 'activation'):
    LOGGING['loggers'][app]['handlers'].append('console')

Now error gets logged both to Airbrake and to the console.