如何使用 requests_html 异步获取（） URL 列表？

Question

I'm trying to asynchronously get() a list of URLs using python package resuqests_html , similar to the async example in the README using Python 3.6.5 and requests_html 0.10.0.

My understanding is that AsyncHTMLSession.run() is supposed work very much the same as asyncio.gather(): You give it a bunch of awaitables, and it runs all of them. Is that incorrect?

Here's the code I'm trying, which I expect should get the pages and store the responses:

from requests_html import AsyncHTMLSession

async def get_link(url):
    r = await asession.get(url)
    return r

asession = AsyncHTMLSession()
results = asession.run(get_link("http://google.com"), get_link("http://yahoo.com"))

But I'm getting this exception instead:

Traceback (most recent call last):
  File "test.py", line 10, in <module>
    results = asession.run(get_link("google.com"), get_link("yahoo.com"))
  File ".\venv\lib\site-packages\requests_html.py", line 772, in run
    asyncio.ensure_future(coro()) for coro in coros
  File ".\venv\lib\site-packages\requests_html.py", line 772, in <listcomp>
    asyncio.ensure_future(coro()) for coro in coros
TypeError: 'coroutine' object is not callable
sys:1: RuntimeWarning: coroutine 'get_link' was never awaited

Am I doing something wrong?

Answer 1

Am I doing something wrong?

You are not calling asession.run correctly.

asyncio.gather accepts awaitable objects, such as coroutine objects obtained by just calling a coroutine (async) function. asession.run , on the other hand, accepts callables , such as async functions, which it will invoke to produce awaitables. The difference is like between one function that accepts an iterable, and which you could pass eg an instantiated generator, and another that accepts a callable that will return an iterable, and which you could pass a generator function itself.

Since your async functions have arguments, you cannot just pass get_link to asession.run ; you must use functools.partial or a lambda itself:

results = asession.run(
    lambda: get_link("http://google.com"),
    lambda: get_link("http://yahoo.com"),
)