Python Requests库在日志记录中抛出异常

Question

The Python requests library appears to have some rather strange quirks when it comes to its logging behaviour. Using the latest Python 2.7.8, I have the following code:

import requests
import logging

logging.basicConfig(
    filename='mylog.txt',
    format='%(asctime)-19.19s|%(task)-36s|%(levelname)s:%(name)s: %(message)s',
    level=eval('logging.%s' % 'DEBUG'))

logger = logging.getLogger(__name__)

logger.info('myprogram starting up...', extra={'task': ''})     # so far so good
...
(ommited code)
...
payload = {'id': 'abc', 'status': 'ok'}

# At this point the program continues but throws an exception.
requests.get('http://localhost:9100/notify', params=payload) 

print 'Task is complete! NotifyURL was hit! - Exiting'

My program seems to exit normally, however inside the log file it creates (mylog.txt) I always find the following exception:

KeyError: 'task'
Logged from file connectionpool.py, line 362

If I remove this: requests.get('http://localhost:9100/notify', params=payload) then the exception is gone.

What exactly am I doing wrong here and how may I fix this? I am using requests v2.4.3.

Answer 1

The problem is your custom logging format, where you expect a %(task) . Requests (or rather the bundled urllib3) does not include the task parameter when logging, as it has no way of knowing, that you expect this.

Answer 2

As indicated in t-8ch 's answer , the logger is being used internally by the requests library and this library doesn't know anything about your parameters. A possible solution is to implant a custom filter in the library's logger (in this case, one of its modules):

class TaskAddingFilter(logging.Filter):
    def __init__(self):
        logging.Filter.__init__(self)

    def filter(self, record):
        record.args = record.args + ('task', '')

# ...

requestsLogger = logging.getLogger('requests.packages.urllib3.connectionpool')
requestsLogger.addFilter(TaskAddingFilter())

Potentially, you need to add such filtering to all loggers from requests , which are:

• requests.packages.urllib3.util
• requests.packages.urllib3.connectionpool
• requests.packages
• requests.packages.urllib3
• requests.packages.urllib3.util.retry
• requests
• requests.packages.urllib3.poolmanager

in my version, you can find them using the logging.Logger.manager.loggerDict attribute. So, you could do something like this:

for name,logger in logging.Logger.manager.loggerDict.iteritems():
    logger = logging.getLogger(name) # because of lazy initialization
    if name.startswith('requests.'):
        logger.addFilter(TaskAddingFilter())

The TaskAddingFilter can be a bit smarter of course, eg adding a particular task entry depending on where you are in your code. I've added only the simplest solution for the code you've provided - the exception doesn't occur anymore - but the wide range of possibilities seems obvious ;)