python aiohttp性能：在主线程上执行连接

Question

I have the following code

import asyncio
import aiohttp

urls = [
    'http://54.224.27.241',
    'http://54.224.27.241',
    'http://54.224.27.241',
    'http://54.224.27.241',
    'http://54.224.27.241',
]

async def query(urls):
    out = []
    with aiohttp.ClientSession() as session:
        for url in urls:
            try:
                async with session.get(url, timeout=5) as resp:
                    text = await resp.text()
                    out.append(resp.status)
            except:
                print('timeout')
    return out


loop = asyncio.get_event_loop()
out = loop.run_until_complete(query(urls))
loop.close()

print(str(out))

The code is much slower than the one that uses a threadpool and keep increasing if you increase the number of urls (lets say 20, 50 etc.)

I have a feeling that when the initial connection establishment is not done in an async way.

(Note that I am connecting here to an non-existing server to deliberately produce a connection timeout).

Can someone point out what is wrong here?

Answer 1

Warning: I don't promise this code works, as I can't install aiohttp atm, but looking at the example in the docs

async def fetch(session, url):
    async with async_timeout.timeout(10):
        async with session.get(url) as response:
            return await response.text()

async def main():
    async with aiohttp.ClientSession() as session:
        html = await fetch(session, 'http://python.org')
        print(html)

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

Notice how they're calling the aiohttp.ClientSession() with the async keyword. Additionally, I was getting some error in your line data = await async with session.get(url) as resp:

async def fetch(session, url):
    async with session.get(url) as response:
        return await response.text()

async def main():
    out = []
    async with aiohttp.ClientSession() as session:
        for url in urls:
            data = await fetch(session, url)
            out.append(data)
    return out

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())