Custom Python logging formatter works with fileConfig but not with dictConfig?

Question

I've been pulling my hair out for hours trying to figure out why these two logging configs, which as far as I can tell should be 100% identical, are not resulting in the same behavior:

LOGGING_CONFIG = {
    'version': 1,
    'disable_existing_loggers': False,
    'root': {
        'handlers': ['console'],
        'level': 'INFO',
    },
    'gunicorn.access': {
        'level': 'INFO',
        'handlers': ['access_console'],
        'propagate': False,
        'qualname': 'gunicorn.access',
    },
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
            'stream': 'ext://sys.stderr',
            'formatter': 'syslog',
        },
        'access_console': {
            'class': 'logging.StreamHandler',
            'stream': 'ext://sys.stdout',
            'formatter': 'docker_access',
        },
    },
    'formatters': {
        'syslog': {
            '()': 'citi.logging.DockerFormatter',
            'fmt': 'SYSLOG %(asctime)s [%(levelname)s] %(name)s: %(message)s',
        },
        'docker_access': {
            '()': 'citi.logging.DockerFormatter',
            'fmt': 'GUNICORN_ACCESS %(asctime)s %(message)s',
        },
    },
}

[loggers]
keys=root,gunicorn.access

[handlers]
keys=console,access_console

[formatters]
keys=syslog,docker_access

[logger_root]
level=INFO
handlers=console

[logger_gunicorn.access]
level=INFO
handlers=access_console
propagate=0
qualname=gunicorn.access

[handler_console]
class=StreamHandler
formatter=syslog
args=(sys.stderr, )

[handler_access_console]
class=StreamHandler
formatter=docker_access
args=(sys.stdout, )

[formatter_syslog]
class=citi.logging.DockerFormatter
format=SYSLOG %(asctime)s [%(levelname)s] %(name)s: %(message)s

[formatter_docker_access]
class=citi.logging.DockerFormatter
format=GUNICORN_ACCESS %(asctime)s %(message)s

When I use logging.config.fileConfig and pass it the file that contains the second code block, it works fine. The gunicorn.access logger changes its formatter to the custom DockerFormatter class (which is a simple subclass of logging.Formatter ) with the specified format string. The access logs look like this:

GUNICORN_ACCESS 2021-07-16 18:59:56,454 172.18.0.1 - - "GET /citi/server-status HTTP/1.0" 200 11 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 11766

But when I use logging.config.dictConfig and pass it the dict at the top, everything except the custom formatter works. The formatter appears to remain set to the original defaults provided by Gunicorn (maybe??). The output is:

172.18.0.1 - - "GET /citi/server-status HTTP/1.0" 200 11 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 7549

By using the logging_tree package, I can tell that gunicorn.access 's formatter is some sort of default when using dictConfig . It should look like this:

   o<--"gunicorn"
   |   Level NOTSET so inherits level INFO
   |   |
   |   o   "gunicorn.access"
   |   |   Level INFO
   |   |   Propagate OFF
   |   |   Handler Stream <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
   |   |     Formatter <citi.logging.DockerFormatter object at 0x7f68240e7d90>

But instead it looks like this:

   o<--"gunicorn"
   |   Level NOTSET so inherits level INFO
   |   |
   |   o   "gunicorn.access"
   |   |   Level INFO
   |   |   Propagate OFF
   |   |   Handler Stream <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
   |   |     Formatter fmt='%(message)s' datefmt=None

The only difference I can detect in terms of how the DockerFormatter class is called is that the fileConfig form calls it with positional arguments, while dictConfig calls it with (correct) keyword arguments.

What appears to be happening is that the formatter is being constructed correctly, but not assigned correctly. And I can't figure out WHY.

The most infuriating thing is that I've actually simplified my logging config quite a bit. In my full config, I set up a custom structlog -based formatter, configured in exactly the same manner as syslog and docker_access are with DockerFormatter , but it DOES work. I am very frustrated. >_<

Answer 1

And of course, my final, last ditch effort before leaving the office for the day (several hours late...) actually works. Turns out I was right about Gunicorn's default settings staying in place, because I had completely missed the absolutely essential 'loggers' subdict in my dictConfig dictionary.

It's supposed to look this THIS:

LOGGING_CONFIG = {
    'version': 1,
    'disable_existing_loggers': False,
    'root': {
        'handlers': ['console'],
        'level': 'INFO',
    },
    'loggers' {
        'gunicorn.access': {
            'level': 'INFO',
            'handlers': ['access_console'],
            'propagate': False,
            'qualname': 'gunicorn.access',
        },
    }
   ...