Log rotation - python and windows

Question

(I have searched and not found a duplicate for this question but happy to be proved otherwise).

I need to rotate a log from within some Python code. The code is running on Windows (Server 2008 R2).

Initially I used TimedRotatingFileHandler (from Python's logging.handlers package) but this doesn't work as we need due to what I understand is an issue it has with multi-processing (subprocess.check_call is used to kick off another application).

I have checked out ConcurrentLogHandler which looks like it might do the job but I'm a bit concerned that it hasn't been updated since 2013 (although issues have been raised more recently).

UPDATE : an open bug (since 2013) indicates that ConcurrentLogHandler does not work with Python 2.7/Windows. On logging, the code just hangs.

Is there a best practice Windows solution I should be using?

Answer 1

Maybe I'm missing something, but Python's logging module comes with a RotatingFileHandler :

import logging
import time

from logging.handlers import RotatingFileHandler

#----------------------------------------------------------------------
def create_rotating_log(path):
    """
    Creates a rotating log
    """
    logger = logging.getLogger("Rotating Log")
    logger.setLevel(logging.INFO)

    # add a rotating handler
    handler = RotatingFileHandler(path, maxBytes=20,
                                  backupCount=5)
    logger.addHandler(handler)

    for i in range(10):
        logger.info("This is test log line %s" % i)
        time.sleep(1.5)

#----------------------------------------------------------------------
if __name__ == "__main__":
    log_file = r"c:\path\to\test.log"
    create_rotating_log(log_file)

This worked fine for me with Python 2.7 on Windows 7. Here are a couple of links that go into more detail:

• http://www.blog.pythonlibrary.org/2014/02/11/python-how-to-create-rotating-logs/
• https://docs.python.org/2/library/logging.handlers.html#rotatingfilehandler

Answer 2

OK - this is what I ended up doing.

Because as far as the logging is concerned things are only multithreaded (additional process does not write to my log) I have ended up hand cranking this. I'm not thrilled by this approach - create a threading lock object, close the logging (logging.shutdown - not thrilled about that either, as the doco says to call it on program exit ...), move the file & start up logging again. Then release the lock.

It is all in a try/except block so that if something goes wrong the lock is released.

Testing indicates that this does what is required.

Does calling logging.shutdown in this kind of context have some repercussions I'm not aware of?!

Answer 3

QueueHandler , which is addressed in the comment of the original question, is available from Python 3.2.

• https://docs.python.org/ko/3/library/logging.handlers.html#queuehandler
• https://docs.python.org/3/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes

Python documentation also suggests using SocketHandler to send all logs into a socket server that dedicates the file writing process.

• https://docs.python.org/3/howto/logging-cookbook.html#network-logging