在单个线程中使用阻塞HTTP库和Python并发http请求

Question

So I have this problem I am trying to solve in a particular way, but I am not sure how hard it is to achieve.

I would like to use the asyncio/coroutines features of Python 3.4 to trigger many concurrent http requests using a blocking http library, like requests, or any Python api that does http requests like boto for aws.

I know about run_in_executor() method to run tasks in threads/processes, but I would like to avoid that.

I would like to do it in a single-thread, using those select features in Linux/Unix kernel.

Actually I was following David Beazley's presentation on this, and I was trying to use this code: https://github.com/dabeaz/concurrencylive/blob/master/aserver.py

but without the future/pool stuff, and use my blocking-api call instead of computing the Fibonacci number.

Put it seems that the http requests are still running in sequence.

Any ideas if this is possible? And how?

Thanks

Answer 1

Not possible. All the calls that the requests library makes to the underlying socket are blocking (ie socket.read ) because the socket is in blocking mode. You could put the socket into non-blocking mode, but then socket.read would fail. You basically need an event-loop to tell you when it's possible to do a socket.read , but blocking libraries aren't written with one in mind. This is the whole reason why asyncio exists; providing a default event-loop that different libraries can share and make use of non-blocking file descriptors (eg sockets).

Use aiohttp , it's just as easy as requests and in the process you get to learn more about asyncio . asyncio and the new Python 3.5 async/await syntax are the Future of networking IO; yield to it (pun intended).