asyncio.gather 没有等待足够长的时间来完成所有任务

Question

I'm writing a code to get some links from a list of input urls using asyncio, aiohttp and BeautifulSoup.

Here's a snippet of the relevant code:

def async_get_jpg_links(links):
    def extractLinks(ep_num, html):
        soup = bs4.BeautifulSoup(html, 'lxml', 
            parse_only = bs4.SoupStrainer('article'))
        main = soup.findChildren('img')
        return ep_num, [img_link.get('data-src') for img_link in main]

    async def get_htmllinks(session, ep_num, ep_link):
        async with session.get(ep_link) as response:
            html_txt = await response.text()
        return extractLinks(ep_num, html_txt)

    async def get_jpg_links(ep_links):
        async with aiohttp.ClientSession() as session:
            tasks = [get_htmllinks(session, num, link) 
                    for num, link in enumerate(ep_links, 1)]
            return await asyncio.gather(*tasks)

    loop = asyncio.get_event_loop()
    return loop.run_until_complete(get_jpg_links(links))

I then later call jpgs_links = dict(async_get_jpg_links(hrefs)) , where hrefs is a bunch of links (~170 links).

jpgs_links should be a dictionary with numerical keys and a bunch of lists as values. Some of the values come back as empty lists (which should instead be filled with data). When I cut down the numbers of links in hrefs , more of the lists come back full.

For the photo below, I reran the same code with a minute between, and as you can see, I get different lists that come back empty and different ones that come back full.

Could it be that asyncio.gather is not waiting for all the tasks to finish?

How can I get asyncio to get me to return no empty lists, while keeping the number of links in hrefs high?

Answer 1

So, turns out that some of the urls I sent in threw up the error:

raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 504, message='Gateway Time-out',...

So I changed

async def get_htmllinks(session, ep_num, ep_link):
        async with session.get(ep_link) as response:
            html_txt = await response.text()
        return extractLinks(ep_num, html_txt)

to

async def get_htmllinks(session, ep_num, ep_link):
    html_txt = None
    while not html_txt:
        try:
            async with session.get(ep_link) as response:
                response.raise_for_status()
                html_txt = await response.text()
        except aiohttp.ClientResponseError:
            await asyncio.sleep(1)
    return extractLinks(ep_num, html_txt)

What this does is that it retries the connection after sleeping for a second (the await asyncio.sleep(1) does that).

Nothing to do with asyncio or BeautifulSoup, apparently.