Optimizing Python logging code

Question

I have a Python application written on the Pyramid framework.

We really take advantage of logging on this ( for debugging ) , using the standard import library.

As we profile stuff on production, it seems that there is quite a bit of overhead from our logging activity. All the string formatting and loops add up. I'd love to remove it, but we can't -- we do actually need to keep it in for testing and will sometimes need to debug on the production environment.

I'm wondering if anyone has effective strategies for minimizing logging as needed , so this code could get "optimized away" during execution on our production environment and simply not run .

For example, under mod_perl, the compiler would "optimize away" statements run under False constants

in psuedocode... ( i haven't touched perl in a long time! )

use constant DEBUG => False ;
if ( DEBUG ) {
      log.debug("stuff here " + string );
}

or

use constant DEBUG => False ;
DEBUG && log.debug("stuff here " + string );

Under those scenarios, the call to log.debug and even the string interpolation would never happen.

Can anyone recommend an effective method of mimicking that behavior under Python ?

Answer 1

Don't use concatenation when you can use log.debug('stuff here %s', string) ; the logging module postpones interpolation until actually formatting the string when logging. If the DEBUG log level has been disabled, no interpolation takes place.

You can also test the logging level to avoid gathering expensive logging information unless needed:

if logger.isEnabledFor(logging.DEBUG):
    logger.debug('Message with %s, %s', expensive_func1(),
                                        expensive_func2())

See the optimization section of the Logging HOWTO .

Answer 2

Use __debug__ . This flag is set to False when Python is run with the -O command line flag, and Python will furthermore optimize out debug code at compile time. I wrote a blog post on it a couple months ago.

Answer 3

I suggest to use assert for that purpose, as it nicely is optimized away in optimized mode.

In this case, it is not necessary to manually check if the logging level is enabled. This is especially useful when doing expensive formatting operations that can be useful for logging (like with the pprint module).

An example:

Instead of

logging.debug('my message %s', pprint.pformat(mycomplexdata))

you would write

assert(logging.debug('my message %s', pprint.pformat(mycomplexdata)) or True)

(The part or True makes sure the assertion does not fail.)

It is even possible to use .format in this case without performance penalty when running with -O .

assert(logging.debug('my message {}'.format(pprint.pformat(mycomplexdata))) or True)

Everything in assert() is optimized away when running python with -O .