Python 日志与性能

Question

I'm using python Logger in one of my programs.

The program is a solver for an np-hard problem and therefore uses deep iterations that run several times.

My question is if the Logger can be an issue in the performance of my program and if there are better ways to log information maintaining performance.

Answer 1

Depending on your Logger configuration and the amount of logs your program produces, yes, logging can be a performance bottleneck because of the blocking Logger operation. For example when directly logging to an NFS file from a NFS server with slow response time. One possible approach to improve performance in such case would be switching to use of a logserver able to buffer and possibly batch logging operations - the blocking would be limited to the communication with the logserver, not to the (slow) logfile access, which is often better from the performance prospective.

Answer 2

I had very good experience using two different logfiles.

1. The server.log file is for the operator and receives only important messages, usually INFO, WARNING, ERROR, CRITICAL.
2. The debug.log file for the developer to analyze errors. It contains up to 100 DEBUG message from the thread of the time before an ERROR occurred.

For the second file, I use thread-local ring-buffers that are only written to a file when the program detects an error. Thus the server.log file remains small but the developers get enough debug messages to analyze problems later. If not problem occurs, then both files are totally empty, and thus do not harm the performance. Of course, the buffers cost memory and a little CPU power, but that can be accepted.

This is an example implementation which I am using in Odoo (which is a Python application):

import logging, collections, time

class LogBuffer(logging.Handler):
    """Buffer debug messages per thread and write them out when an error (or warning) occurs"""

    def __init__(self, target_handler, threshold, max_buffered_messages, max_buffer_seconds):
        logging.Handler.__init__(self, logging.DEBUG)
        self.tread_buffers = dict()  # stores one buffer for each thread (key=thread number)
        self.target_handler = target_handler
        self.threshold = threshold
        self.max_buffered_messages = max_buffered_messages
        self.last_check_time = time.time()
        self.max_buffer_seconds = max_buffer_seconds

    def emit(self, record):
        """Do whatever it takes to actually log the specified logging record."""

        # Create a thread local buffer, if not already exists
        if record.thread not in self.tread_buffers:
            thread_buffer = self.tread_buffers[record.thread] = collections.deque()
        else:
            thread_buffer = self.tread_buffers[record.thread]

        # Put the log record into the buffer
        thread_buffer.append(record)

        # If the buffer became to large, then remove the oldest entry
        if len(thread_buffer) > self.max_buffered_messages:
            thread_buffer.popleft()

        # produce output if the log level is high enough
        if record.levelno >= self.threshold:
            for r in thread_buffer:
                self.target_handler.emit(r)
            thread_buffer.clear()

        # remove very old messages from all buffers once per minute
        now = time.time()
        elapsed = now - self.last_check_time
        if elapsed > 60:
            # Iterate over all buffers
            for key, buffer in list(self.tread_buffers.items()):
                # Iterate over the content of one buffer
                for r in list(buffer):
                    age = now - r.created
                    if age > self.max_buffer_seconds:
                        buffer.remove(r)
                # If the buffer is now empty, then remove it
                if not buffer:
                    del self.tread_buffers[key]
            self.last_check_time = now

Example how to create/configure such a logger:

import logging
from . import logbuffer

"""
    Possible placeholders for the formatter:

    %(name)s            Name of the logger (logging channel)
    %(levelno)s         Numeric logging level for the message (DEBUG, INFO,
                        WARNING, ERROR, CRITICAL)
    %(levelname)s       Text logging level for the message ("DEBUG", "INFO",
                        "WARNING", "ERROR", "CRITICAL")
    %(pathname)s        Full pathname of the source file where the logging
                        call was issued (if available)
    %(filename)s        Filename portion of pathname
    %(module)s          Module (name portion of filename)
    %(lineno)d          Source line number where the logging call was issued
                        (if available)
    %(funcName)s        Function name
    %(created)f         Time when the LogRecord was created (time.time()
                        return value)
    %(asctime)s         Textual time when the LogRecord was created
    %(msecs)d           Millisecond portion of the creation time
    %(relativeCreated)d Time in milliseconds when the LogRecord was created,
                        relative to the time the logging module was loaded
                        (typically at application startup time)
    %(thread)d          Thread ID (if available)
    %(threadName)s      Thread name (if available)
    %(process)d         Process ID (if available)
    %(message)s         The result of record.getMessage(), computed just as
                        the record is emitted
"""

# Log levels are: CRITICAL, ERROR, WARNING, INFO, DEBUG

# Specify the output format
formatter = logging.Formatter('%(asctime)-15s %(thread)20d %(levelname)-8s %(name)s %(message)s')

# Create server.log
server_log = logging.FileHandler('../log/server.log')
server_log.setLevel(logging.INFO)
server_log.setFormatter(formatter)
logging.root.addHandler(server_log)

# Create debug.log
debug_log = logging.FileHandler('../log/debug.log')
debug_log.setFormatter(formatter)
memory_handler = logbuffer.LogBuffer(debug_log, threshold=logging.ERROR, max_buffered_messages=100, max_buffer_seconds=600)
logging.root.addHandler(memory_handler)

# Specify log levels for individual packages
logging.getLogger('odoo.addons').setLevel(logging.DEBUG)

# The default log level for all other packages
logging.root.setLevel(logging.INFO)

Please let me know if you find this helpful. Im on a very beginner level regarding Python, but I have the same thing in Java and C++ already running successfully for years.