How to configure python's logging module to use the formatter for multiline output?

Question

I've configured the logging module to format log messages with:

...
    'formatters': {
        'verbose': {
            'format': '%(asctime)s [%(levelname)s] '
                      '(%(name)s:pid=%(process)d) %(module)s(%(lineno)d): '
                      '%(message)s'
        },
    },
...

However, when it prints an exception, it shows up like:

2014-02-12 14:16:20,454 [ERROR] (app:pid=1930) main(105): Catchall exception at top level.
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/argh/dispatching.py", line 228, in dispatch_command
    dispatch(parser, *args, **kwargs)
  File "build/bdist.linux-x86_64/egg/argh/dispatching.py", line 124, in dispatch
    for line in lines:
  File "build/bdist.linux-x86_64/egg/argh/dispatching.py", line 200, in _execute_command
    for line in result:
  ...,
NameError: global name 'sftp' is not defined

Is it possible to make it run each line through the formatter? Eg so it would output something like:

2014-02-12 14:16:20,454 [ERROR] (app:pid=1930) main(105): Catchall exception at top level.
2014-02-12 14:16:20,454 [ERROR] (app:pid=1930) main(105): Traceback (most recent call last):
2014-02-12 14:16:20,454 [ERROR] (app:pid=1930) main(105):   File "build/bdist.linux-x86_64/egg/argh/dispatching.py", line 228, in dispatch_command
2014-02-12 14:16:20,454 [ERROR] (app:pid=1930) main(105):     dispatch(parser, *args, **kwargs)
2014-02-12 14:16:20,454 [ERROR] (app:pid=1930) main(105):   File "build/bdist.linux-x86_64/egg/argh/dispatching.py", line 124, in dispatch
2014-02-12 14:16:20,454 [ERROR] (app:pid=1930) main(105):     for line in lines:
2014-02-12 14:16:20,454 [ERROR] (app:pid=1930) main(105):   File "build/bdist.linux-x86_64/egg/argh/dispatching.py", line 200, in _execute_command
2014-02-12 14:16:20,454 [ERROR] (app:pid=1930) main(105):     for line in result:
2014-02-12 14:16:20,454 [ERROR] (app:pid=1930) main(105):   ...,
2014-02-12 14:16:20,454 [ERROR] (app:pid=1930) main(105): NameError: global name 'sftp' is not defined

Answer 1

You can do this without configuration. Use traceback.format_exec to get the stack trace as a string, then split the lines and log each line individually.

import logging
import traceback

logger = logging.Logger(__name__) # or whatever namespace you so please
# do any other Logger configuration you want

try:
    do_some_stuff()
except Exception as e:
    for line in traceback.format_exec().split('\n'):
        logger.error(line)

This is not common practice, however. If you use Logger.exception() you could log the exception. It would be on one logging line, but it would be written on multiple lines to the file-like object to which the Logger is writing.

(Of interest is the fact that the logging.Formatter class' method, formatException , uses traceback.print_exception for its default implementation.)

Answer 2

I think you can use json module for that

import json

a = {"a": 1, "b": 2}

print json.dumps(a, indent=4)

output:

{
    "a": 1, 
    "b": 2
}