Python Requests library throws exceptions in logging

Question

The Python requests library appears to have some rather strange quirks when it comes to its logging behaviour. Using the latest Python 2.7.8, I have the following code:

import requests
import logging

logging.basicConfig(
    filename='mylog.txt',
    format='%(asctime)-19.19s|%(task)-36s|%(levelname)s:%(name)s: %(message)s',
    level=eval('logging.%s' % 'DEBUG'))

logger = logging.getLogger(__name__)

logger.info('myprogram starting up...', extra={'task': ''})     # so far so good
...
(ommited code)
...
payload = {'id': 'abc', 'status': 'ok'}

# At this point the program continues but throws an exception.
requests.get('http://localhost:9100/notify', params=payload) 

print 'Task is complete! NotifyURL was hit! - Exiting'

My program seems to exit normally, however inside the log file it creates (mylog.txt) I always find the following exception:

KeyError: 'task'
Logged from file connectionpool.py, line 362

If I remove this: requests.get('http://localhost:9100/notify', params=payload) then the exception is gone.

What exactly am I doing wrong here and how may I fix this? I am using requests v2.4.3.

Answer 1

The problem is your custom logging format, where you expect a %(task) . Requests (or rather the bundled urllib3) does not include the task parameter when logging, as it has no way of knowing, that you expect this.

Answer 2

As indicated in t-8ch 's answer , the logger is being used internally by the requests library and this library doesn't know anything about your parameters. A possible solution is to implant a custom filter in the library's logger (in this case, one of its modules):

class TaskAddingFilter(logging.Filter):
    def __init__(self):
        logging.Filter.__init__(self)

    def filter(self, record):
        record.args = record.args + ('task', '')

# ...

requestsLogger = logging.getLogger('requests.packages.urllib3.connectionpool')
requestsLogger.addFilter(TaskAddingFilter())

Potentially, you need to add such filtering to all loggers from requests , which are:

• requests.packages.urllib3.util
• requests.packages.urllib3.connectionpool
• requests.packages
• requests.packages.urllib3
• requests.packages.urllib3.util.retry
• requests
• requests.packages.urllib3.poolmanager

in my version, you can find them using the logging.Logger.manager.loggerDict attribute. So, you could do something like this:

for name,logger in logging.Logger.manager.loggerDict.iteritems():
    logger = logging.getLogger(name) # because of lazy initialization
    if name.startswith('requests.'):
        logger.addFilter(TaskAddingFilter())

The TaskAddingFilter can be a bit smarter of course, eg adding a particular task entry depending on where you are in your code. I've added only the simplest solution for the code you've provided - the exception doesn't occur anymore - but the wide range of possibilities seems obvious ;)