Sample logging generation python logging library

Question

I would like to sample the number of logs I send from my service as it has quite some traffic. Let's say I wanna send logs for 10% of the requests using python's standard logging library. Is there a native/elegant way to do it that centralizes this decision for all loggers? I think maybe some sort of configuration in the root logger would do the job but not sure how.

Thank you in advance!

Answer 1

There are many ways to go about this. Personally, I like to create my own Logger class that I call to log instead of directly calling the standard logging library. It gives one a lot more freedom when trying to change or add any functionality, and you have it all in one centralized place; as opposed to having to make the changes in every single file or request handler. If you do that, you'd also be able to add optional parameters to certain log lines, like the frequency (or probability) of logging a certain log line.

But you can also tweak the standard logging, as you requested, to get this done. I will provide both options for clarity:

Option 1: Using Python's logging library

Here, we add a filter to Python's logger (thank you @AKX for the suggestion)

import logging
from random import SystemRandom


class ProbabilityFilter(logging.Filter):
    probability = 0.10
    cryptogen = SystemRandom()

    def filter(self, record):
        return self.cryptogen.random() < self.probability



logger = logging.getLogger('test_logger')
logger.setLevel('INFO')
logger.addFilter(ProbabilityFilter())
logger.addHandler(logging.StreamHandler())


for i in range(100):
    logger.info("Logging number: {}".format(i))

There are some caveats to this solution:

• You will have to add this to every handler you want to add this functionality to, possibly resulting in a lot of changes.
• From what you mentioned, you want to log the requests you're receiving. If you apply this handler to a logger, it will log 10% of all messages it receives. And this includes other messages from the same logger that you may always want to log, instead.

Option 2: Using your own custom logger

from random import SystemRandom
import logging


class MyAppLogger:

    def __init__(self, logger_name: str):
        self.logger = logging.getLogger(logger_name)
        self.logger.setLevel("INFO")
        self.logger.addHandler(logging.StreamHandler())
        self.cryptogen = SystemRandom()

    def info(self, message: str, probability: float = 1.0):
        """Logs a message at an INFO level.

        :param message: The logging message to print.
        :type message: str
        :param probability: The chance of logging the message. Always log
                by default.
        :type probability: float, optional
        """

        # Checks the probability first to avoid the cost of generating
        # a random number if avoidable
        if probability == 1.0 or self.cryptogen.random() < probability:
            self.logger.info(message)


logger = MyAppLogger('test_logger')

# By default, log everything
for i in range(5):
    logger.info("Definitely logging number: {}".format(i))


# Logging with a probability, with the same logger
for i in range(100):
    logger.info("Possibly logging number: {}".format(i), probability=0.1)

You may need to replace all your loggers initially, but this encapsulation gives you a lot more flexibility when it comes to adding parameters or tweaking anything else you want to add or remove in the future.

Hope this helps and have a great day!