Structlog add processor to handler

Question

I'm struggling to set different processors for different handlers in structlog.

Here is a minimal example of what I'm trying to achieve:

import logging
import structlog
import sys

from logging.handlers import TimedRotatingFileHandler


def elastic_format(logger: logging.Logger, method_name: str, event_dict: dict):
    # Elastic requires the message to be under 'message' and not under 'event'
    if isinstance(event_dict, dict) and event_dict.get('event') and not event_dict.get('message'):
        event_dict['message'] = event_dict.pop('event')

    return event_dict

structlog.configure_once(
    # Don't mess up the order of this!
    # Note this https://www.structlog.org/en/stable/standard-library.html
    # On the structlog side, the processor chain must be configured to end with
    # structlog.stdlib.ProcessorFormatter.wrap_for_formatter as the renderer.
    processors=[
        structlog.stdlib.add_log_level,
        structlog.stdlib.add_logger_name,
        structlog.processors.TimeStamper(fmt='iso'),
        structlog.processors.StackInfoRenderer(),
        structlog.processors.format_exc_info,
        structlog.stdlib.ProcessorFormatter.wrap_for_formatter
    ],
    context_class=structlog.threadlocal.wrap_dict(dict),
    logger_factory=structlog.stdlib.LoggerFactory(),
    wrapper_class=structlog.stdlib.BoundLogger,
    cache_logger_on_first_use=True
)

console_handler = logging.StreamHandler(sys.stdout)
console_handler.setFormatter(structlog.stdlib.ProcessorFormatter(processor=structlog.dev.ConsoleRenderer(colors=True)))

file_handler = TimedRotatingFileHandler(filename='./my_logs.log', interval=1, backupCount=48, encoding='utf-8')
file_handler.setFormatter(structlog.stdlib.ProcessorFormatter(processor=structlog.processors.JSONRenderer()))

root_logger = logging.getLogger()
root_logger.addHandler(console_handler)
root_logger.addHandler(file_handler)

root_logger.setLevel('INFO')

logger = structlog.getLogger('my_logger')
logger.info('My log message.', extra={'abc': 123})

The console output is:

2021-12-30T21:55:58.793483Z [info     ] My log message.                [my_logger] extra={'abc': 123}

The file output is:

{"extra": {"abc": 123}, "event": "My log message.", "level": "info", "logger": "my_logger", "timestamp": "2021-12-30T21:55:58.793483Z"}

The above output is what I expect, however, for the file handler, I'd like to add the elastic_format function included in the code example.

I do not see a way to set the processors individually on the handlers. This currently just changes the log data but will apply a few transformations on the data once fully implemented. How can this be achieved using structlog?

To clarify, I want this to apply only to the file handler and not the console handler so I don't believe it can be added to the structlog.configure_once section.

Answer 1

structlog 21.3.0 has added a processors (plural) argument to structlog.stdlib.ProcessorFormatter (as a replacement for processor (singular)).

It takes a full chain so I believe that should allow you to achieve what you want by adding elastic_format there?