Using logging.info instead of logger.info wrecks StreamHandler

Question

I am using two logging handlers. One to a file for level DEBUG and above, and the other to the console for WARNING and above. I'm using many modules, some of them external (installed using pip). When there is a call to logging.info() (as opposed to logger.info() ), this wrecks the settings for the console logger:

import logging
import logging.handlers
import sys, os
from demo_module import something, something_else

logger = logging.getLogger("demo")
logger.setLevel(logging.DEBUG)
main_handler = logging.FileHandler('demo.log')
formatter = logging.Formatter('%(asctime)s %(name)s %(levelname)-8s %(filename)s:%(lineno)d %(message)s')
main_handler.setFormatter(formatter)
logger.addHandler(main_handler)

# log serious issues to console
console_handler = logging.StreamHandler(stream=sys.stderr)
console_handler.setLevel(logging.WARNING)
console_handler.setFormatter(formatter)
logger.addHandler(console_handler)
logger.debug('this is a debug')
logger.info('this is an info')
logger.warning('this is a warning')
logger.error('this is an error')
logger.critical('reactor has melted down')
something()
logger.info('after something')
something_else()
logger.info('after something else')

Where demo_module.py is:

import logging
logger = logging.getLogger('demo')

def something():
    logger.info('something needs to be done')
    a = 1+1
    logger.info('something has been done')
    logger.error('some banale error')

def something_else():
    logger.info('calling logger.info')
    # OOPS, calling logging instead of logger here
    logging.info("module may use logging directly instead of logger")
    logger.info('logger.info called')

As you can see, inside demo_module.something_else() there is a call to logging.info instead of logger.info .

The output to the console looks like this:

% python logger.py 
2017-10-07 17:45:08,077 demo WARNING  logger2.py:21 this is a warning
2017-10-07 17:45:08,077 demo ERROR    logger2.py:22 this is an error
2017-10-07 17:45:08,077 demo CRITICAL logger2.py:23 reactor has melted down
2017-10-07 17:45:08,078 demo ERROR    demo_module.py:8 some banale error
INFO:demo:logger.info called
INFO:demo:after something else

As you can see, the last two lines lose all the settings I originally set to the console handler, like the level, and the formatter. If it was only my code, I could find this offending logging.info() statement and replace it, but some of the code is not mine but third party and as such is beyond my control. Any idea how to make those settings stick?

Answer 1

By default module level logging functions will calling logging.basicConfig() to try to configure the root logger. Docs :

... module-level convenience functions, which delegate to the root logger, call basicConfig() to ensure that at least one handler is available.

logging.basicConfig 's docs state:

This function does nothing if the root logger already has handlers configured for it.

So the solution is to configure the handlers on the root logger before any of the module level log functions get called, while also instructing your logger not to propagate messages up the logger hierarchy, to avoid duplicate output.

import logging
import logging.handlers
import sys, os
from demo import something, something_else

formatter = logging.Formatter('%(asctime)s %(name)s %(levelname)-8s %(filename)s:%(lineno)d %(message)s')

# Configure the root logger.
root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)
# Configure file handler.
root_main_handler = logging.FileHandler('demo.log')
root_main_handler.setFormatter(formatter)
root_logger.addHandler(root_main_handler)
# Configure console handler.
root_console_handler = logging.StreamHandler(stream=sys.stderr)
root_console_handler.setLevel(logging.WARNING)
root_console_handler.setFormatter(formatter)
root_logger.addHandler(root_console_handler)

logger = logging.getLogger("demo")
logger.setLevel(logging.DEBUG)
main_handler = logging.FileHandler('demo.log')
main_handler.setFormatter(formatter)
logger.addHandler(main_handler)

# log serious issues to console
console_handler = logging.StreamHandler(stream=sys.stderr)
console_handler.setLevel(logging.WARNING)
console_handler.setFormatter(formatter)
logger.addHandler(console_handler)
# Don't propagate messages up to the root logger.
logger.propagate = False
logger.debug('this is a debug')
logger.info('this is an info')
logger.warning('this is a warning')
logger.error('this is an error')
logger.critical('reactor has melted down')
something()
logger.info('after something')
something_else()
logger.info('after something else')

Console output:

2017-10-07 18:46:50,644 demo WARNING  run.py:35 this is a warning
2017-10-07 18:46:50,644 demo ERROR    run.py:36 this is an error
2017-10-07 18:46:50,644 demo CRITICAL run.py:37 reactor has melted down
2017-10-07 18:46:50,645 demo ERROR    demo.py:8 some banale error

File output:

2017-10-07 18:46:50,644 demo DEBUG    run.py:33 this is a debug
2017-10-07 18:46:50,644 demo INFO     run.py:34 this is an info
2017-10-07 18:46:50,644 demo WARNING  run.py:35 this is a warning
2017-10-07 18:46:50,644 demo ERROR    run.py:36 this is an error
2017-10-07 18:46:50,644 demo CRITICAL run.py:37 reactor has melted down
2017-10-07 18:46:50,645 demo INFO     demo.py:5 something needs to be done
2017-10-07 18:46:50,645 demo INFO     demo.py:7 something has been done
2017-10-07 18:46:50,645 demo ERROR    demo.py:8 some banale error
2017-10-07 18:46:50,645 demo INFO     run.py:39 after something
2017-10-07 18:46:50,645 demo INFO     demo.py:11 calling logger.info
2017-10-07 18:46:50,645 root INFO     demo.py:13 module may use logging directly instead of logger
2017-10-07 18:46:50,645 demo INFO     demo.py:14 logger.info called
2017-10-07 18:46:50,645 demo INFO     run.py:41 after something else