I'm using the Retry module from Python urllib3
+ requests
for a case where a 3rd party API gives sporadic errors. One issue I have is that if the retries keep failing, I get a exceptions.MaxRetryError
and never get to see what the response was. What if there was valuable debugging data coming from the server?
Is there a way to still get the response in cases that would throw MaxRetryError
?
Here's my code below
from requests.packages.urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter
def req_with_retry(retries=3, backoff_factor=0.5, status_forcelist=(400, 404, 500, 502, 504,), method_whitelist=frozenset(['POST', 'HEAD', 'TRACE', 'GET', 'PUT', 'OPTIONS', 'DELETE']), session=None,):
'''
this returns a session that functions like the requests module but with retries built it for certain status codes
'''
session = session or requests.Session()
retry = Retry(
total=retries,
read=retries,
connect=retries,
backoff_factor=backoff_factor,
status_forcelist=status_forcelist,
method_whitelist=method_whitelist
)
adapter = HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://', adapter)
return session
r = req_with_retry().get(url="https://www.google.com")
First, I would like to mention that from requests.packages.urllib3.util.retry import Retry
is no longer needed since urllib3 has been unvendored from requests. Consider using the from urllib3.util.retry import Retry
form instead as it does not involve hacks (see https://github.com/psf/requests/blob/v2.22.0/requests/packages.py ).
You can't get both the MaxRetryError
and the response from urllib3. But you can get the response from urllib3, and use raise_for_status
at the requests level. However, raise_for_status()
won't be affected by your status_forcelist
, so you may want to reimplement it yourself.
Here's an adaption of your code that 1/ retries on statuses 2/ prints the final response 3/ raises an exception.
import logging
import urllib3
import requests
import requests.adapters
logging.basicConfig(level=logging.DEBUG)
logging.getLogger("urllib3").setLevel(logging.DEBUG)
def req_with_retry(
retries=3,
backoff_factor=0.5,
status_forcelist=(400, 404, 500, 502, 504),
method_whitelist=frozenset(
["POST", "HEAD", "TRACE", "GET", "PUT", "OPTIONS", "DELETE"]
),
session=None,
):
"""Returns a session that functions like the requests module but with retries
built it for certain status codes
"""
session = session or requests.Session()
retry = urllib3.Retry(
total=retries,
read=retries,
connect=retries,
backoff_factor=backoff_factor,
status_forcelist=status_forcelist,
method_whitelist=method_whitelist,
raise_on_status=False,
)
adapter = requests.adapters.HTTPAdapter(max_retries=retry)
session.mount("http://", adapter)
session.mount("https://", adapter)
return session
r = req_with_retry().get(url="https://httpbin.org/status/404")
print(r.status_code, r.text)
r.raise_for_status()
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.