如何使用 httpx.AsyncClient 作为 class 成员，并异步关闭

Question

I want to use http client as a class member, but del function could not call await client.aclose() . eg:

import httpx

class Foo(object):
    def __init__(self):
        self.client = httpx.AsyncClient()

    def __del__(self):
        await self.client.aclose()

refer:https://www.python-httpx.org/async/#opening-and-closing-clients how to safely aclose?

Answer 1

Although this is an older question, I may have something compelling to share as I had a similar situation. To @Isabi's point (Answered 2020-12-28), you need to use an event loop to decouple the client from your operations and then manually control it's lifecycle.

In my case, I need more control over the client such that I can separate the Request from the sending and when the client is closed so I can take advantage of session pooling, etc. The example provided below shows how to use http.AsyncClient as a class member and close it on exit.

In figuring this out, I bumped into an Asyncio learning curve but quickly discovered that it's... actually not too bad. It's not as clean as Go[lang] but it starts making sense after an hour or two of fiddling around with it. Full disclosure: I still question whether this is 100% correct.

The critical pieces are in the __init__ , close , and the __del__ methods. What, to me, remains to be answered, is whether using a the http.AsyncClient in a context manager actually resets connections, etc. I can only assume it does because that's what makes sense to me. I can't help but wonder: is this even necessary?

import asyncio
import httpx
import time
from typing import Callable, List
from rich import print


class DadJokes:

    headers = dict(Accept='application/json')

    def __init__(self):
        """
        Since we want to reuse the client, we can't use a context manager that closes it.
        We need to use a loop to exert more control over when the client is closed.  
        """
        self.client = httpx.AsyncClient(headers=self.headers)
        self.loop = asyncio.get_event_loop()

    async def close(self):
        # httpx.AsyncClient.aclose must be awaited!
        await self.client.aclose()

    def __del__(self):
        """
        A destructor is provided to ensure that the client and the event loop are closed at exit.
        """
        # Use the loop to call async close, then stop/close loop.
        self.loop.run_until_complete(self.close())
        self.loop.close()

    async def _get(self, url: str, idx: int = None):
        start = time.time() 
        response = await self.client.get(url)
        print(response.json(), int((time.time() - start) * 1000), idx)

    def get(self, url: str):
        self.loop.run_until_complete(self._get(url))

    def get_many(self, urls: List[str]):
        start = time.time()
        group = asyncio.gather(*(self._get(url, idx=idx) for idx, url in enumerate(urls)))
        self.loop.run_until_complete(group)
        print("Runtime: ", int((time.time() - start) * 1000))


url = 'https://www.icanhazdadjoke.com'
dj = DadJokes()
dj.get_many([url for x in range(4)])

Since I've been using Go as of late, I originally wrote some of these methods with closures as they seemed to make sense; in the end I was able to (IMHO) provide a nice balance in between separation / encapsulation / isolation by converting the closures to class methods.

The resulting usage interface feels approachable and easy to read - I see myself writing class based async moving forward.

Answer 2

The problem might be due to the fact that client.aclose() returns an awaitable , which cannot be awaited in a normal def function.

It could be worth giving a try with asyncio.run(self.client.aclose()) . Here it might occur an exception, complaining that you are using a different event loop (or the same, I don't know much of your context so I can't tell) from currently running one. In this case you could get the currently running event loop and run the function from there.

See https://docs.python.org/3/library/asyncio-eventloop.html for more information on how you could accomplish it.