![](/img/trans.png)
[英]How to retry the task when aiohttp.ClientSession fails in async
[英]how to retry async aiohttp requests depending on the status code
我正在使用一個 api,有時它會給出一些奇怪的狀態代碼,可以通過簡單地重試相同的請求來修復這些代碼。 我正在使用 aiohttp 異步地向這個 api 發出請求
我也在使用退避庫來重試請求,但是似乎仍然沒有重試 401 請求。
@backoff.on_exception(backoff.expo, aiohttp.ClientError, max_tries=11, max_time=60)
async def get_user_timeline(self, session, user_id, count, max_id, trim_user, include_rts, tweet_mode):
params = {
'user_id': user_id,
'trim_user': trim_user,
'include_rts': include_rts,
'tweet_mode': tweet_mode,
'count': count
}
if (max_id and max_id != -1):
params.update({'max_id': max_id})
headers = {
'Authorization': 'Bearer {}'.format(self.access_token)
}
users_lookup_url = "/1.1/statuses/user_timeline.json"
url = self.base_url + users_lookup_url
async with session.get(url, params=params, headers=headers) as response:
result = await response.json()
response = {
'result': result,
'status': response.status,
'headers': response.headers
}
return response
如果響應的狀態代碼不是 200 或 429,我希望所有請求最多可退出 10 次。
我做了一個簡單的庫,可以幫助你:
https://github.com/inyutin/aiohttp_retry
這樣的代碼應該可以解決您的問題:
from aiohttp import ClientSession
from aiohttp_retry import RetryClient
statuses = {x for x in range(100, 600)}
statuses.remove(200)
statuses.remove(429)
async with ClientSession() as client:
retry_client = RetryClient(client)
async with retry_client.get("https://google.com", retry_attempts=10, retry_for_statuses=statuses) as response:
text = await response.text()
print(text)
await retry_client.close()
相反google.com
使用您自己的url
默認情況下,aiohttp 不會引發非 200 狀態的異常。 您應該通過raise_for_status=True
( doc ) 更改它:
async with session.get(url, params=params, headers=headers, raise_for_status=True) as response:
它應該為任何 400 或更高的狀態引發異常,從而觸發backoff
。
代碼 2xx 不應該重試,因為這些不是錯誤。
無論如何,如果您仍然想為“非 200 或 429”加注,您可以手動進行:
if response.status not in (200, 429,):
raise aiohttp.ClientResponseError()
也許它太舊了,但是對於任何想知道如何構建這樣的解決方案的人來說
RequestData 和 ErrorResponseData 是您的自定義類,它不是內置的
class DataAPI:
def __init__(self, api_data_converter: APIDataConverter):
self.api_data_converter = api_data_converter
async def _bound_fetch(self, request_data: RequestData, session):
try:
async with session.get(request_data.url, raise_for_status=True) as response:
return ResponseData(await response.text())
except aiohttp.ClientConnectionError as e:
Logging.log_exception('Connection error: {}'.format(str(e)))
return ErrorResponseData(url=request_data.url, request_data=request_data)
except Exception as e:
Logging.log_exception('Data API error: {}'.format(str(e)))
return ErrorResponseData(url=request_data.url, request_data=request_data)
async def _run_requests(self, request_data: List[RequestData]):
for rd in request_data:
Logging.log_info('Request: {}'.format(rd.url))
async with aiohttp.ClientSession(timeout=ClientTimeout(total=80)) as session:
tasks = []
for rd in request_data:
task = asyncio.ensure_future(self._bound_fetch(rd, session))
tasks.append(task)
responses = asyncio.gather(*tasks)
return await responses
def get_data(self, request_data: List[RequestData]):
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
skipped = request_data
responses: List[ResponseData] = []
for _ in range(2): # specify your retry count instead of 2
interm_responses = loop.run_until_complete(asyncio.ensure_future(self._run_requests(skipped)))
skipped = []
for resp in interm_responses:
if isinstance(resp, ErrorResponseData):
skipped.append(resp.request_data)
else:
responses.append(resp)
if not skipped:
break
if skipped:
Logging.log_critical('Failed urls remaining')
for resp in responses:
data = self.api_data_converter.convert(resp.response)
if not data:
Logging.log_exception('Data API error')
dt = dateutil.parser.parse(data[-1]['dt'])
resp.response = data
resp.last_candle_dt = dt
return responses
Mikhail 的回答涵蓋了如何為狀態代碼 4XX 和 5XX 引發異常,但是如果您希望針對這些狀態代碼重試協程,請查看async_retrying
庫 - https://pypi.org/project/async_retrying/
下面給出一個簡單的例子。
import asyncio
from aiohttp import ClientSession
from async_retrying import retry
@retry(attempts=2)
async def hit_url(url, session: ClientSession):
async with session.get(url) as response:
print("Calling URL : %s", url)
await response.text()
return response.status
async def main():
urls = [
"https://google.com",
"https://yahoo.com"
]
api_calls = []
async with ClientSession(raise_for_status=True) as session:
for url in urls:
api_calls.append(hit_url(session=session, url=url))
await asyncio.gather(*api_calls, return_exceptions=False)
asyncio.run(main())
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.