如何重试python请求库中由对等连接错误重置的104连接而不尝试除外

Question

我有以下 Session object：

    import requests
    from requests.adapters import Retry, HTTPAdapter
    from requests.sessions import Session

    retry_strategy: Retry = Retry(
        total=5,
        status_forcelist=[429, 500, 502, 503, 504],
        allowed_methods=["HEAD", "GET", "OPTIONS", "POST"],
        backoff_factor=1
    )
    http_adapter = HTTPAdapter(max_retries=retry_strategy)
    session = requests.Session()
    session.mount("https://", http_adapter)

有时服务器似乎会断开与 RST TCP 数据包的连接，从而导致以下错误：

{"isError": true,
 "type": "ConnectionError",
 "message": "[Errno 104] Connection reset by peer",
 "traceback": "...  omitted for brevity ...
    File "/var/task/requests/sessions.py", line 590, in post
    return self.request('POST', url, data=data, json=json, **kwargs)    
    File "/var/task/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)   
    File "/var/task/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs) 
    File "/var/task/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)"
    

1. 是否可以在Session object 中注入Retry object 以便自动重试此特定的ConnectionError ？
2. 有没有其他方法可以自动重试而无需任何其他自定义逻辑，例如将其包装在try... except语句中或创建自定义HTTPAdapter ？
3. 我看过多篇关于104 connection reset by peer errors帖子，但没有人询问如何自动重试。 这是为什么？ 我想我不会是唯一遇到这个问题的人吧？

Answer 1

看看https://github.com/jd/tenacity

我通常做这样的事情

from tenacity import wait_exponential

@retry(wait=wait_exponential(multiplier=1, min=4, max=10))
    def wait_exponential_1():
        # my request here

您可以创建一个函数，该函数产生一个用坚韧包装的会话。

Answer 2

我最终以退避告终：

@backoff.on_exception(
    wait_gen=backoff.expo,
    exception=requests.ConnectionError,
    max_time=time_units.in_s.SEC_15,
)
def robust_session_request(*args, **kwargs) -> requests.Response:
    return session.request(*args, **kwargs)

Answer 3

您需要提供“connect”参数：

  from requests import adapters
  import urllib3

  sync_session = requests.Session()
  retry_adapter = adapters.HTTPAdapter(max_retries=urllib3.Retry(total=1, connect=1))
  session.mount('https://', retry_adapter)
  session.mount('http://', retry_adapter)

测试：

class AdapterTest(unittest.TestCase):

  @mock.patch.object(urllib3.connectionpool.HTTPConnectionPool, '_make_request',
                     side_effect=ConnectionResetError)
  def test_adapter(self, mocked):

    s = requests_extensions.HookableSession()
    try:
      s.mount('http://', adapters.HTTPAdapter(max_retries=urllib3.Retry(total=1, connect=1)))
      s.request('GET', 'http://www.test.com')
    except requests.ConnectionError:
      pass

    self.assertEqual(mocked.call_count, 2)

Answer 4

使用这个答案，我创建了一个服务器，它一直返回“连接由对等方重置”。 在一个终端上启动服务器：

$ python server.py

并在另一个终端上测试它：

$ curl http://localhost:5500
curl: (56) Recv failure: Connection reset by peer

服务器很好地显示尝试了一个连接：

$ python server.py 
new client from 127.0.0.1:55894

现在基于这个答案，让我们尝试使用 Python 和重试：

>>> import requests
>>> import urllib3.util
>>> import requests.adapters
>>> s = requests.Session()
>>> retries = urllib3.util.Retry(connect=2, read=3, redirect=4, status=5, other=6)
>>> s.mount('http://', requests.adapters.HTTPAdapter(max_retries=retries))
>>> s.get("http://localhost:5500")

并且服务器显示 4 次连接尝试（假设您在上述 curl 测试后重新启动它）- 1 次初始尝试和 3 次read重试。

$ python server.py 
new client from 127.0.0.1:60790
new client from 127.0.0.1:60798
new client from 127.0.0.1:60810
new client from 127.0.0.1:60818

您也可以只使用total ：

>>> import requests
>>> import urllib3.util
>>> import requests.adapters
>>> s = requests.Session()
>>> retries = urllib3.util.Retry(total=5)
>>> s.mount('http://', requests.adapters.HTTPAdapter(max_retries=retries))
>>> s.get("http://localhost:5500")

$ python server.py 
new client from 127.0.0.1:37164
new client from 127.0.0.1:37176
new client from 127.0.0.1:37190
new client from 127.0.0.1:37192
new client from 127.0.0.1:42852
new client from 127.0.0.1:42864