RuntimeWarning：协程从未在单独的线程中等待 asyncio.run

Question

I've built a little aiohttp based spider. Here's some abbreviated code:

import asyncio
from aiohttp import ClientSession
from threading import Thread

class Spider:
    def __init__(self, urls):
        self.urls = urls
        self.start()

    async def fetch(self, session, url):
        async with session.get(url) as response:
            await self.handle_status(response) # undefined here for brevity
            return await self.render_body(response) # undefined here for brevity

    async def process_urls(self):
        async with ClientSession() as session:
            tasks = {self.fetch(session, url) for url in self.urls}
            for task in asyncio.as_completed(tasks):
                raw_data = await task
                data = self.extract_data(*raw_data) # sync method undefined here for brevity
                await self.store_data(data) # undefined here for brevity

    def start(self) -> None:
        try:
            asyncio.run(self.process_urls())
        except RuntimeError:  # loop already running
            x = Thread(target=asyncio.run, args=(self.process_urls(),))
            x.start()
            x.join()

The start method is intended to start an asyncio loop, but if one is already running, then it starts a new loop in a new Thread.

The code works whether run from an existing loop or not. But if run from an existing loop (eg with pytest.mark.asyncio()), I get this warning:

RuntimeWarning: coroutine 'Spider.process_urls' was never awaited x.join()

Is this warning something I should be concerned about?

Is there a better way to handle this that would not cause this warning?

I've tried loop = asyncio.get_running_loop() with loop.create_task(self.process_urls()) instead of creating a new thread, but this fails with:

RuntimeError: coroutine ignored GeneratorExit

I've tried nest-asyncio, but my primary use case involves another library that uses uvloop, so nest-asyncio is not compatible.

Answer 1

I admit, I can't offer you full explanation on the reasons why that warning is there but I can offer you code that is warning free:

def start(self) -> None:
    try:
        asyncio.get_running_loop()
    except RuntimeError:
        asyncio.run(self.process_urls())
    else:
        # Since the loop is already instantiated we can use it with our own
        # tasks without creating another thread.
        # asyncio.create_task(self.process_urls())
        x = Thread(target=asyncio.run, args=(self.process_urls(),))
        x.start()
        x.join()

Possibly, you get RuntimeWarning because your code creates new Thread (and BTW, that warning gets produced even when you switch to asyncio.create_task() ) during handling of the exception where "normal laws of physics" don't apply. But again, that's only my guess so take it with a grain of salt.