How to add keyword arguments to methods called through ThreadPoolExecuter and run_in_executor?

Question

I am trying to send a POST request concurrently with the help of concurrent.futures . For some reason, I am unable to set custom headers. I want to set

1. Authorization
2. Content-type

Here is the progress that I have made till now.

import asyncio
import concurrent.futures
import requests
from urllib.parse import urlencode, quote_plus


params = urlencode({'a': 'b', 'c': 'd', 'e': 'f'})
headers = {"Content-type": "application/x-www-form-urlencoded","Accept": "*/*","Authorization": "Bearer kdjalskdjalskd"}

async def main():

    with concurrent.futures.ThreadPoolExecutor(max_workers=20) as executor:

        loop = asyncio.get_event_loop()
        futures = [
            loop.run_in_executor(
                executor, 
                requests.post,
                'https://fac03c95.ngrok.io',params, headers)
            for i in range(20)
        ]
        for response in await asyncio.gather(*futures):
            print(response.text)
            pass

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

But for some reason, the headers don't seem to show up in the request. Can anyone help me out here?

Thanks in advance :)

Answer 1

Disclaimer: It's solution of particular problem described in question. Improving quality of provided code is not an object of this answer.

Let's check docs of loop.run_in_executor() and requests.post() .

run_in_executor() pass provided arguments to function. Now let's take a look on your code:

loop.run_in_executor(executor, requests.post, 'https://fac03c95.ngrok.io', params, headers)

So as a result function will be called like this:

requests.post('https://fac03c95.ngrok.io', params, headers)

Let's merge values of provided arguments with their keywords:

• url - https://fac03c95.ngrok.io ;
• data - {'a': 'b', 'c': 'd', 'e': 'f'}
• json - {"Content-type": "application/x-www-form-urlencoded", ...}

To add custom headers you should pass keyword argument headers= . Unfortunately, run_in_executor() doesn't forward keyword arguments, so you have to use some kind of proxy function. Here is few variants:

1. Function .

    def proxy_post(url, data, headers): return requests.post(url, data=data, headers=headers) ... loop.run_in_executor(executor, proxy_post, 'https://fac03c95.ngrok.io', params, headers)

2. lambda .

    loop.run_in_executor(executor, lambda: requests.post('https://fac03c95.ngrok.io', data=params, headers=headers))

3. functools.partial() .

    import functools ... loop.run_in_executor(executor, functools.partial(requests.post, 'https://fac03c95.ngrok.io', data=params, headers=headers))

Answer 2

The bug lies in the way you pass in parameters for the requests.post call:

loop.run_in_executor(
    executor, 
    requests.post, 
    'https://fac03c95.ngrok.io',
    params, 
    headers
)

The requests post request expects keyword arguments for additional parameters such as headers as it seems from the function header . Now unfortunately the loop.run_in_executor call only accepts *args but not keyword arguments (see documentation ) which makes it slightly more tricky to get it work correctly.

Personally, I would just write a little wrapper function for the requests call. It could look something like the following:

def make_request(url, data, headers):
    return requests.post(url, data=data, headers=headers)

And call make_request instead of request.post from your asynchronous call. If we integrate it into your overall script, it could therefore looks something like the following:

import asyncio
import concurrent.futures
import requests
from urllib.parse import urlencode, quote_plus


params = urlencode({'a': 'b', 'c': 'd', 'e': 'f'})
headers = {
    "Content-type": "application/x-www-form-urlencoded",
    "Accept": "*/*",
    "Authorization": "Bearer kdjalskdjalskd"
}

def make_request(url, data, headers):
    return requests.post(url, data=data, headers=headers)

async def main():

    with concurrent.futures.ThreadPoolExecutor(max_workers=20) as executor:

        loop = asyncio.get_event_loop()
        futures = [
            loop.run_in_executor(
                executor, 
                make_request, 
                'https://fac03c95.ngrok.io', 
                params, 
                headers
            )
            for i in range(20)
        ]

        for response in await asyncio.gather(*futures):
            print(response.text)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

Of course, you can add *args to make_request and pass in arguments from your function call - just keep in mind that only regular arguments are supported, not keyword arguments. Any function you wish to execute through your event loop that requires keyword arguments will need to be called through a wrapper.

---

Recommended Approach in the Documentation

The recommended way according to the concurrent.futures documentation to pass in keyword arguments to func is to use functools.partial (as also pointed out in the other answer).

An example utilising functools.partial would look as follows.

from functools import partial

loop.run_in_executor(executor, partial(requests.post, data=params, headers=headers)

Whilst the result is the same as using a wrapper function, the advantage of using partial is that reduced duplication of two function calls into one as we don't need to pass the arguments through a wrapper function anymore.