日志不要覆盖

Question

I'm using Python's logging.config module to configure and use a logging tool in my project.

I want my log files to overwrite each time (not append), so I set my YAML configuration file like this:

# logging configuration file

version: 1
disable_existing_loggers: False

formatters:
    simple:
        format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"

handlers:
    console:
        class: logging.StreamHandler
        level: DEBUG
        formatter: simple
        stream: ext://sys.stdout

    info_file_handler:
        class: logging.handlers.RotatingFileHandler
        level: INFO
        formatter: simple
        filename: .logs/info.log
        maxBytes: 10485760 # 10MB
        backupCount: 20
        encoding: utf8
        mode: 'w'

    error_file_handler:
        class: logging.handlers.RotatingFileHandler
        level: ERROR
        formatter: simple
        filename: .logs/errors.log
        maxBytes: 10485760 # 10MB
        backupCount: 20
        encoding: utf8
        mode: 'w'

loggers:
    my_module:
        level: ERROR
        handlers: [console]
        propagate: no

root:
    level: INFO
    handlers: [console, info_file_handler, error_file_handler]

This question mentions that using mode: w in the handler configuration should accomplish what I want, but the log files keep appending anyway. Here is my configuration code as well:

def logging_setup(cfg_path=definitions.LOG_CONFIG_PATH, def_lvl=logging.INFO):
    """Setup logging tool from YAML configuration file."""

    # create directory for log files if not already there
    try:
        os.makedirs(definitions.LOGS_PATH)
    except OSError as e:
        if e.errno != errno.EEXIST:
            raise

    # configure logging from yaml config file
    if os.path.exists(cfg_path):
        with open(cfg_path, 'rt') as f:
            config = yaml.load(f.read())

        logging.config.dictConfig(config)

    else:
        logging.basicConfig(level=def_lvl)

What am I doing incorrectly?

Answer 1

Invoke doRollover() of the handler object will reset the file handler to overwrite old files before each run.

Python Logging Handlers - Rotating File Handler

Answer 2

logging.handlers.RotatingFileHandler forces append mode by design. See logging/handlers.py :

# If rotation/rollover is wanted, it doesn't make sense to use another
# mode. If for example 'w' were specified, then if there were multiple
# runs of the calling application, the logs from previous runs would be
# lost if the 'w' is respected, because the log file would be truncated
# on each run.
if maxBytes > 0:
    mode = 'a'

Therefore, if you are only interested in the latest run, but have a lot of output that means a single log file is unmanageable, you need to delete the old log files yourself first:

def clean_up_log_files(logfile):
    """
    Cleans up the log file base_file_name and any rotated files.

    RotatingFileHandler is forced into append mode if the log size is specified.
    This means we can't rely on it to clean up our log files, and we have to do
    it for ourselves.

    Args:
        logfile     The log file name passed to RotatingFileHandler.
    """
    if os.path.exists(logfile):
        os.unlink(logfile)
    pattern = logfile + ".*"
    for file in glob(pattern):
        os.unlink(file)

Answer 3

I orginally used logging.handlers.RotatingFileHandler as the original handler class for my logger configuration. I did so because I am trying to minimize possible memory risks on a very small disk. However: using logging.handlers.RotatingFileHandler instead of logging.FileHandler seems to disable the mode: 'w' option in the YAML configuration file. The asker in the linked question seems to suggest he thinks this might be the issue, but the posted answers do not reflect that (thus my confusion).

Using logging.FileHandler fixes my issue, allowing my log files to be overwritten each execution. I understand overwriting logs & rotating logs serve purposes that are in some amount of conflict with one another, but I thought having the safety that rotating logs offer with the desired convenience of overwritten logs would be best for my project.