Python: Log to multiple log files

Question

Currently I have everything getting logged to one logfile but I want to separate it out to multiple log files. I look at the logging in python documentation but they don't discuss about this.

log_format = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
logging.basicConfig(filename=(os.path.join(OUT_DIR, + '-user.log')),
            format=log_format, level=logging.INFO, datefmt='%Y-%m-%d %H:%M:%S')

Currently this is how I do the logging. what I want to do have different type of errors or information get log into different log files. At the moment when I do logging.info('Logging IN') and logging.error('unable to login') will go to same logfile. I want to seperate them. Do I need to create another logging object to support the logging into another file?

Answer 1

What you /could/ do (I haven't dug into the logging module too much so there may be a better way to do this) is maybe use a stream rather than a file object:

In [1]: class LogHandler(object):
   ...:     def write(self, msg):
   ...:         print 'a :%s' % msg
   ...:         print 'b :%s' % msg
   ...:         

In [3]: import logging
In [4]: logging.basicConfig(stream=LogHandler())
In [5]: logging.critical('foo')
a :CRITICAL:root:foo
b :CRITICAL:root:foo

In [6]: logging.warn('bar')
a :WARNING:root:bar
b :WARNING:root:bar

Edit with further handling :

Assuming your log files already exist, you could do something like this:

import logging

class LogHandler(object):
    format = '%(levelname)s %(message)s'
    files = { 
        'ERROR': 'error.log',
        'CRITICAL': 'error.log',
        'WARN': 'warn.log',
    }   
    def write(self, msg):
        type_ = msg[:msg.index(' ')] 
        with open(self.files.get(type_, 'log.log'), 'r+') as f:
            f.write(msg)

logging.basicConfig(format=LogHandler.format, stream=LogHandler())
logging.critical('foo')

This would allow you to split your logging into various files based on conditions in your log messages. If what you're looking for isn't found, it simply defaults to log.log .

Answer 2

I created this solution from docs.python.org/2/howto/logging-cookbook.html

Simply create two logging file handlers, assign their logging level and add them to your logger.

import os
import logging

current_path = os.path.dirname(os.path.realpath(__file__))

logger = logging.getLogger('simple_example')
logger.setLevel(logging.DEBUG)

#to log debug messages                               
debug_log = logging.FileHandler(os.path.join(current_path, 'debug.log'))
debug_log.setLevel(logging.DEBUG)

#to log errors messages
error_log = logging.FileHandler(os.path.join(current_path, 'error.log'))
error_log.setLevel(logging.ERROR)

logger.addHandler(debug_log)
logger.addHandler(error_log)

logger.debug('This message should go in the debug log')
logger.info('and so should this message')
logger.warning('and this message')
logger.error('This message should go in both the debug log and the error log')