Requests in Asyncio - Keyword Arguments

Question

I'm using asyncio with the requests module to make an asynchronous HTTP request.

I can make a GET request like this:

@asyncio.coroutine
def do_checks():
    loop = asyncio.get_event_loop()
    req = loop.run_in_executor(None, requests.get, 'https://api.github.com/user')
    resp = yield from req
    print(resp.status_code)
loop = asyncio.get_event_loop()
loop.run_until_complete(do_checks())

However, I need to do support Basic HTTP Auth ( described here ) in the request.

According to the documentation, url and auth are both named parameters for requests.get().

But, if I run this (note the addition of url='' and auth = '' ):

@asyncio.coroutine
def do_checks():
    loop = asyncio.get_event_loop()
    req = loop.run_in_executor(None, requests.get, url='https://api.github.com/user', auth=HTTPBasicAuth('user', 'pass'))
    resp = yield from req
    print(resp.status_code)
loop = asyncio.get_event_loop()
loop.run_until_complete(do_checks())

I get this error:

TypeError: run_in_executor() got an unexpected keyword argument 'url'

In the prototype for asyncio.run_in_executor(), additional arguments are supported:

BaseEventLoop.run_in_executor(executor, callback, *args)

requests.get() clearly supports named parameters (get, auth, etc.). What's wrong?

Answer 1

Two ways to do that. Create a wrapper function, or just use a session to provide the auth.

Using a session:

@asyncio.coroutine
def do_checks():
    loop = asyncio.get_event_loop()
    session = requests.Session()
    session.auth = HTTPBasicAuth('user', 'pass')
    req = loop.run_in_executor(None, session.get, 'https://api.github.com/user')
    resp = yield from req
    print(resp.status_code)

Writing a wrapper function (note that I'm using def for clarity here, but that a lambda would obviously work too):

@asyncio.coroutine
def do_checks():
    def do_req():
        return requests.get('https://api.github.com/user', auth=HTTPBasicAuth('user', 'pass'))
    loop = asyncio.get_event_loop()
    req = loop.run_in_executor(None, do_req)
    resp = yield from req
    print(resp.status_code)

Answer 2

This is actually a design decision in asyncio . From the docstring of asyncio/base_events.py:

"""Base implementation of event loop.

The event loop can be broken up into a multiplexer (the part
responsible for notifying us of IO events) and the event loop proper,
which wraps a multiplexer with functionality for scheduling callbacks,
immediately or at a given time in the future.

Whenever a public API takes a callback, subsequent positional
arguments will be passed to the callback if/when it is called.  This
avoids the proliferation of trivial lambdas implementing closures.
Keyword arguments for the callback are not supported; this is a
conscious design decision, leaving the door open for keyword arguments
to modify the meaning of the API call itself.
"""

Note the last sentence there.

The asyncio PEP notes this as well, and recommends a lambda to work around it:

This convention specifically does not support keyword arguments. Keyword arguments are used to pass optional extra information about the callback. This allows graceful evolution of the API without having to worry about whether a keyword might be significant to a callee somewhere. If you have a callback that must be called with a keyword argument, you can use a lambda. For example:

loop.call_soon(lambda: foo('abc', repeat=42))