Rewrite a library with blocking requests into async ones

Question

There is a library that uses blocking requests in its core and I would like to rewrite it into asynchronous version, so could you please advise what would be the best/easiest strategy to do so.

The whole library, after several nested functions, calls one function:

def _send_http_request(self, url, payload, method='post', **kwargs):
# type: (Text, Optional[Text], Text, dict) -> Response

response = request(method=method, url=url, data=payload, **kwargs)

return response

Just putting async in front of it wont work since it is deeply nested in blocking functions. And rewriting everything would be a way too much hassle. I had a look into aiohttp , trio , asks and kinda got lost, which one is better. I know about celery or dask , but I need async.

Answer 1

You have several options:

1. Rewrite _send_http_request to be async (using, for example, aiohttp ) and further rewrite all functions that use _send_http_request to be async either. Yes, it's much work to do, but this is how asyncio fundamentally designed.

2. Wrap only top-level blocking functions (functions with I/O) you need to run asynchronously with run_in_executor as explained here . If you aren't going to make millions of requests you won't see much performance difference with option above since main bottleneck is still I/O. Otherwise threads overhead will be noticeable compared to pure asyncio solution.

3. Try other solution instead of asyncio . For example, gevent and its monkey-patching . This approach has own pros and cons.