Python logging: debug messages logged to stderr even though handler level is INFO

Question

While modifying code on an existing project, I found that debug messages were being output to stderr even though the handler level for these logs is set to INFO level so I do not expect any debug messages. The cause of the behavior seems to be that an import is using logging.debug() where after that call it changed the way messages were being output to the console.

I reduced the issue to the code example below to get help on understanding why these debug level messages are being logged:

import logging
import sys

# get root logger singleton
root_logger = logging.getLogger()

# create my own logger object
test_logger = logging.getLogger("test_logger")
test_logger.setLevel(logging.DEBUG)

# add handler to my logger
sh = logging.StreamHandler(sys.stdout)
sh.setLevel(logging.INFO)
test_logger.addHandler(sh)

test_logger.info("Info message from test_logger")

test_logger.debug("Debug message before logging.debug")
print("test_logger handlers before: " + str(test_logger.handlers))
print("test_logger level before: " + str(test_logger.getEffectiveLevel()))
print("Root before: " + str(root_logger.handlers) + " level: "+ str(root_logger.getEffectiveLevel()))

# This call causes the debug messages after it to go to stderr
# Also, it results in the root logger object having a handler added to it 
logging.debug("Debug message from logging")

# This debug message is unexpectedly logged to stderr
test_logger.debug("Debug message after logging.debug")
print("test_logger handlers after: " + str(test_logger.handlers))
print("test_logger logging level after: " + str(test_logger.getEffectiveLevel()))
print("Root after: " + str(root_logger.handlers) + " level: " + str(root_logger.getEffectiveLevel()))

The code above has the following output, where for some reason only one of the debug messages is output and is going to stderr:

DEBUG:test_logger:Debug message after logging.debug
Info message from test_logger
test_logger handlers before: [<StreamHandler <stdout> (INFO)>]
test_logger level before: 10
Root before: [] level: 30
test_logger handlers after: [<StreamHandler <stdout> (INFO)>]
test_logger logging level after: 10
Root after: [<StreamHandler <stderr> (NOTSET)>] level: 30

It makes sense to me why the StreamHandler is being added to the root logger since logging.debug() is called and with no handler on root it will add one. The diagram at https://docs.python.org/3.5/howto/logging.html#logging-flow also seems relevant in that it shows that messages from loggers with a parent go to the parent and the root logger should be the parent of test_logger.

Adding the following line of code will prevent the debug message from being generated in stderr so it seems the issue has to do with the message being propagated to its ancestors. However, I am not sure that is an appropriate solution or just a workaround.

test_logger.propagate = False

The docs on this at https://docs.python.org/3/library/logging.html#logging.Logger.propagate explain that:

Messages are passed directly to the ancestor loggers' handlers - neither the level nor filters of the ancestor loggers in question are considered.

To me that means the root logger should get this message but the debug message that is printed shows the logger as test_logger DEBUG:test_logger:Debug message after logging.debug so it doesn't appear to be coming from the root logger. The root logger also has the default level of logging.WARN (30) and my understanding is that if a handler of a logger processes a debug message, it is only printed if the log level on the logger is equal or lower to the message level?

Is anyone able to explain what is going on here and if possibly I am missing some mandatory step? For example, in this code should I be initializing the root logger to avoid default handlers being added even if I create a named logger and plan to use that?

UPDATE: As @blues pointed out, this is working as expected and I was not reading the documentation correctly. The messages are propagated directly to parent logger handlers not the actual loggers.

I stepped through the Python logging library (v3.7.3) to demonstrate this behavior in the code, as it seemed odd to me at first.

The callHandlers() method of this stack trace snippet shows how this happens.

callHandlers, init .py:1585

handle, init .py:1529

_log, init .py:1519

debug, init .py:1371

Answer 1

It seems you are misreading the documentation. When messages are propagated they are not propagated to the parent logger. They are propagated directly to the handlers of the parent. It is not actually the case but it helps to think about it like this: propagation adds the handlers of the parent to the child logger. Since your test_logger has a level of DEBUG and the direct call to logging.debug() causes a handler with NOTSET level being added to root which is a parent of test_logger the situation is as if test_logger had this handler which will log any messages, so all debug messages sent to the test_logger will be logged by this handler.