Keep aiohttp session alive

Question

I'm trying to visit a website every X seconds with parallel and separate sessions, then analyse what is in the response to see if each session should continue or not. However, once the code reaches the second loop it fails.

import asyncio
from aiohttp import ClientSession
import logging
import time

interval = 30
instances = 2
visit_url = 'http://www.example.org'

tasks = []

logging.basicConfig(
    format='%(asctime)s.%(msecs)03d %(message)s',  # Log in format time.milliseconds {message}
    level=logging.INFO,  # Use with logging.info()
    datefmt='%H:%M:%S')  # Display time as Hours:Minutes:Seconds


class StopException(Exception):
    pass


async def quit_app(session, task_, reason):
    logging.info("[{}] {}.".format(task_, reason))
    session.cookies.clear()  # Reset cookies
    session.headers.clear()  # Reset headers
    session.close()  # End HTTP connection
    raise StopException


async def get_status(response):
    if "abow" in response:
        return "success"
    elif "odoap" or "daoscp" in response:
        return "waiting"
    elif "nullt" in response:
        return "fail"
    elif "issue" in response:
        return "banned"
    elif "pending" in response:
        return "pending"
    else:
        return "pending"


async def initialise(headers):
    session = ClientSession()
    task_ = len(asyncio.Task.all_tasks()) - instances - 1
    passed = False
    while passed is False:
        async with session as session:
            async with session.get(visit_url, headers=headers, allow_redirects=True) as initial:
                status = await get_status(await initial.text())  # Check HTML for status
                if status == "success":
                    logging.info("[{}] {}.".format(task_, "Success"))
                    passed = True
                elif status == "pending":
                    logging.info("[{}] {}.".format(task_, "Pending.."))
                    await asyncio.sleep(interval)
                elif status == "waiting":
                    logging.info("[{}] {}.".format(task_, "Waiting..."))
                    await asyncio.sleep(interval)
                elif status == "banned":
                    await quit_app(initial, task_, "Banned")
                elif status == "fail":
                    await quit_app(initial, task_, "Failed")


if __name__ == "__main__":
    headers = {
        'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
        'accept-encoding': 'gzip, deflate, br',
        'accept-language': 'en-US,en;q=0.9',
        'upgrade-insecure-asks': '1',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36'
    }  # Add appropriate headers
    start = time.clock()
    loop = asyncio.get_event_loop()
    for i in range(instances):
        task = asyncio.ensure_future(initialise(headers))
        tasks.append(task)
    loop.run_until_complete(asyncio.wait(tasks))
    end = time.clock()
    print("Process took {0:.4f} seconds.".format(end - start))

This code returns the following error:

13:56:58.604 Task exception was never retrieved future: Task finished coro= initialise() done, defined at C:/Users/x/PycharmProjects/tests/src/aiohttp_main.py:49 exception=RuntimeError('Session is closed',) RuntimeError: Session is closed

I just can't work out how to make the sessions stay alive until I .close() them...

Answer 1

I just can't work out how to make the sessions stay alive until I .close() them...

The async with is a request to close the session. Which means that after you write:

async with session as session:

...you can no longer use session once the body of async with is done executing. This is not specific to aiohttp, it's how with works in Python. For example, when working with files, with is a request to close the file:

with open('data.csv') as fileobj:
    # ... read stuff from fileobj

# outside the "with" block, fileobj is closed and you
# can no longer read from it

The fix is simple enough, just move the with outside the while loop. For example:

async def initialise(headers):
    async with ClientSession() as session:
        # the rest of the code, including the `while` loop, here

On an unrelated note, you probably want to replace len(asyncio.Task.all_tasks()) with a global counter of your own. Using Task.all_tasks() in this way can start producing incorrect results if you later incorporate other unrelated tasks into the event loop (or a third-party library does that for you).