python logging setting multiple loggers and switching between them

Question

I have a program with a loop, and I would like to set up a main logger which logs information from the whole program. I would like to set up a different logger inside a loop (I will call this a 'logger_instance' going forward) for each loop instance. The main logger also needs to be able to log some information within a loop.

The issue with my current code is the following: once I initialize a logger_instance inside a loop, the information that I intended to log to the main logger gets logged to the logger_instance instead of the main logger.

Below is the sample code:

class DispatchingFormatter:
"""
This allows to create several formatter objects with desired formats so that logging outputs can take different
formats
"""
    def __init__(self, formatters, default_formatter):
        self._formatters = formatters
        self._default_formatter = default_formatter

    def format(self, record):
        formatter_obj = self._formatters.get(record.name, self._default_formatter)
        return formatter_obj.format(record)

def initiate_logger(log_file_name):

    # Set logging to display INFO level or above
    logging.basicConfig(level=logging.INFO)

    # First empty out list of handlers to remove the default handler created by the running basicConfig above
    # logging.getLogger().handlers.clear()
    logger = logging.getLogger()
    logger.handlers.clear()
    # logger = logging.getLogger().handlers.clear()

    # Set up formatter objects
    formatter_obj = DispatchingFormatter(
    # Custom formats - add new desired formats here
    {
        # This format allows to print the user and the time - use this log format at the start of the execution
        'start_log': logging.Formatter('\n%(levelname)s - %(message)s executed the pipeline at %(asctime)s',
                                       datefmt='%Y-%m-%d %H:%M:%S'),
        # This format allows to print the time - use this log format at the end of the execution
        'end_log': logging.Formatter('\n%(levelname)s - pipeline execution completed at %(asctime)s',
                                     datefmt='%Y-%m-%d %H:%M:%S'),
        # This format allows to print the duration - use this log format at the end of the execution
        'duration_log': logging.Formatter('\n%(levelname)s - total time elapsed: %(message)s minutes'),
        # This is the format of warning messages
        'warning_log': logging.Formatter('\n\t\t%(levelname)s - %(message)s'),
        # This is the format of error messages (
        'error_log': logging.Formatter('\n%(levelname)s! [%(filename)s:line %(lineno)s - %(funcName)20s()] - '
                                       'Pipeline terminating!\n\t%(message)s')
    },
        # Default format - this default is used to print all ESN outputs
        logging.Formatter('%(message)s'),
    )

    # Log to console (stdout)
    c_handler = logging.StreamHandler()
    c_handler.setFormatter(formatter_obj)
    # logging.getLogger().addHandler(c_handler)
    logger.addHandler(c_handler)

    # Log to file in runs folder
    f_handler = logging.FileHandler(log_file_name, 'w+')
    f_handler.setFormatter(formatter_obj)
    # logging.getLogger().addHandler(f_handler)

    logger.addHandler(f_handler)

    # Log user and start time information upon creating the logger
    logging.getLogger('start_log').info(f'{getpass.getuser()}')

    return logger

if __name__ == '__main__':
    # Test logging

    # Initial main logger for outside the loop
    logger_main = initiate_logger('log_main.txt')
    logger_main.info(f'Started the main logging')

    for i in range(5):
        # Create logger in a loop
        this_logger = initiate_logger(f'log_{str(i)}.txt')
        this_logger.info(f'Logging in a loop - loop #{str(i)}')

        # Log some information to main logger
        logger_main.info(f'Something happened in loop #{str(i)}')

    # Log more information to main logger after the loop
    logger_main.info(f'Loop completed!')

log_main.txt contains

INFO - this_user executed the pipeline at 2019-05-29 19:15:47
Started the main logging

log_0.txt contains

INFO - lqk4061 executed the pipeline at 2019-05-29 19:15:47
Logging in a loop - loop #0
Something happened in loop #0

Desired output for log_main.txt should be

INFO - this_user executed the pipeline at 2019-05-29 19:15:47
Started the main logging
Something happened in loop #0
Something happened in loop #1
Something happened in loop #2
Something happened in loop #3
Something happened in loop #4
Loop completed!

Desired output for log_0.txt should be

INFO - lqk4061 executed the pipeline at 2019-05-29 19:15:47
Logging in a loop - loop #0

Any help will be greatly appreciated!

Answer 1

That happens because your initiate_logger function always returns the root logger since it calls getlogger without a name. See the documentation . What you need to do is give each of them a different name if you want them to be different logger instances. for example logger = logging.getLogger(log_file_name) would work in your code. I would recomment using filters instead though.