Python multithreading for writing log

Question

I have an HTTP Server class, supporting multithreading, that while writing back the result to the client, it writes some logs (Logging module) as well. It slows down the request, and since the POST/GET methods returns only after the 'return' occures, I have to write the log before i send the result. I wish to do it by opening another thread that will receive the message, and write the log. I wish to have my Logging object as 'static', or a class member which is one and only, and is being shared to all threads that can acquire a lock on it and use it.

import logging

LOG_FILENAME = 'babyServer.log'
class Logger(object):
    logging.basicConfig(level=logging.DEBUG,
                    format='%(asctime)s %(levelname)-8s %(message)s',
                    datefmt='%a, %d %b %Y %H:%M:%S',
                    filename='serverLog.log',
                    filemode='wa')


    @classmethod
    def Write(cls,level,log):
        level = level.lower()
        if level == 'error':
            logging.error(log)
        if level == 'info':
            logging.info(log)
        if level == 'exception':
            logging.exception(log)

So from somewhere outside of this file, I have currently a call such as Logger.Write('info', "some info") Instead, I wish that this call will actually open a new thread, that owns a reference to the single Logging I configured in the Logger class.

Answer 1

Starting a thread also has overhead.

So I would suggest that you write a module that has three public functions;

• start(filename) to start a logging thread
• write(txt) to write to the log file.
• stop() to stop the logging thread.

The write function places the text to log on the FIFO Queue and returns to the calling thread.

The logging thread reads items from the Queue and writes them to the logfile.

Say you have a file log.py like this;

"""Module to log to a file."""

import Queue
import threading

_name = None
_lt = None
_q = None
_stop = False


def _run():
    while not _stop:
        txt = _q.get()
        with open(_name, 'a+') as logfile:
            logfile.write(txt)


def start(name):
    global _lt, _q, _name
    if _name:
        raise RuntimeError('Logging already initialized')
    _name = name
    _q = Queue.Queue()
    _lt = threading.Thread(target=_run)
    _lt.start()


def stop():
    global _stop
    _stop = True
    _lt.join()


def write(txt):
    _q.put(txt, block=False)

Using the logging module instead of simply wrting to a file is left as an exercise for the reader.

Answer 2

I found a very simple solution, using thread module.

import thread

Then, When I need to call my logging function, I simply use it with the following:

thread.start_new_thread( logging.error, (log,) )

where in this example, logging.error is a builtin function of logging, making an error log, and (log,) is a tuple (as it is required by the interface) of my the log message to write. It, as I expected, improve performance dramatically

    @classmethod
def Write(cls,level,log):
    level = level.lower()
    if level == 'error':
        thread.start_new_thread( logging.error, (log,) )
    if level == 'info':
        thread.start_new_thread( logging.info, (log,) )
    if level == 'exception':
        thread.start_new_thread( logging.exception, (log,) )