协同之外的Aiohttp ClientSession

Question

I have a REST API wrapper which is supposed to run in an interactive Python session. HTTP requests are made both through an automated background thread (which uses the API wrapper) and manually by the end user through the interactive session. I am trying to migrate all HTTP request management to asyncio from the former new-thread-per-request approach, but since I can't run the asyncio loop in the main thread (it has to be free for ad-hoc Python commands/requests), I wrote the following to run it in a background thread:

import aiohttp
import asyncio
from concurrent.futures import ThreadPoolExecutor

def start_thread_loop(pool=None):
    """Starts thread with running loop, bounding the loop to the thread"""
    def init_loop(loop):
        asyncio.set_event_loop(loop)  # bound loop to thread
        loop.run_forever()
    _pool = ThreadPoolExecutor() if pool is None else pool
    loop = asyncio.new_event_loop()
    future = _pool.submit(init_loop, loop)
    return future, loop

def send_to_loop(coro, loop):
    """Wraps couroutine in Task object and sends it to given loop"""
    return asyncio.run_coroutine_threadsafe(coro, loop=loop)

The actual API wrapper is something like the following:

class Foo:
    def __init__(self):
        _, self.loop = start_thread_loop()
        self.session = aiohttp.ClientSession(loop=self.loop)
        self.loop.set_debug(True)

    def send_url(self, url):
        async def _request(url):
            print('sending request')
            async with self.session.get(url) as resp:
                print(resp.status)
        return send_to_loop(_request(url), self.loop)

However, aiohttp highly recommends against making a ClientSession outside of coroutine and turning on asyncio debug mode before initializing ClientSession raises a RuntimeError . Therefore, I tried making a slightly different version using asycio.Queue in order to avoid making a ClientSession inside a coroutine:

class Bar:

    def __init__(self):
        _, self.loop = start_thread_loop()
        self.q = asyncio.Queue(loop=self.loop)
        self.status = send_to_loop(self.main(), loop=self.loop)

    async def main(self):
        async with aiohttp.ClientSession(loop=self.loop) as session:
            while True:
                url = await self.q.get()
                print('sending request')
                asyncio.ensure_future(self._process_url(url, session), loop=self.loop)

    def send_url(self, url):
        send_to_loop(self.q.put(url), loop=self.loop)

    @staticmethod
    async def _process_url(url, session):
        async with session.get(url) as resp:
            print(resp.status)

However, this approach is more complex/verbose and I don't really understand if it is actually necessary.

Questions:

1. Why is it a problem starting a ClientSession outside of a coroutine?
2. Is the queue approach better/safer? If so, why?
3. Is there any problem in my approach for starting a loop inside a background thread?

Answer 1

Why is it a problem starting a ClientSession outside of a coroutine?

That's how aiohttp is built, in theory it should be possible to initialize some kind of client session outside a loop ie. outside a coroutine but it's not how aiohttp is built. AFAIU in the issue that introduced this warning , it's because a) it's difficult to test b) it's prone to error

Is the queue approach better/safer? If so, why?

I don't understand what you try to achieve so I am not sure how to answer. Maybe the problem you have is that you try to initialize a ClientSession inside the constructor aka. __init__ of another class. In this case, you should work around that issue by creating a helper method that is a coroutine that will finish the initialization of the class. This is known pattern when working with async code.

Is there any problem in my approach for starting a loop inside a background thread?

It's perfectly ok.