POST 多个请求 concurrent.futures Status = Pending, Error = Timeout

Question

问题：

我有大量产品数据需要通过 REST API 发布到我的网站。

数据：

[{'name': '101300-92010 TOOL, LINER UPPER',
  'description': '101300-92010 TOOL, LINER UPPER',
  'short_description': '<font size="+3">\r\nFits Engines:\r\n</font>NFAD6-(E)KMK\r\n NFAD6-KMK2\r\n NFAD7-KMK\r\n NFAD7-LEIK3\r\n NFAD7-LIK3\r\n NFAD8-(E)KMK\r\n NFD9-(E)KMK\r\n TF50\r\n TF55H-DI\r\n TF55L\r\n TF55R-DI\r\n TF60\r\n TF60B\r\n TF65H-DI\r\n TF65L\r\n TF65R-DI\r\n TF70\r\n TS105C-GBB\r\n TS190R-B\r\n TS60C-GB\r\n TS70C-GB\r\n TS80C-GBB',
  'tags': [{'name': '101300-92010'}],
  'sku': '98JCDOZE4P',
  'categories': [{'id': 307931}],
  'slug': '101300-92010-tool-liner upper-yanmar/'},

 {'name': '103288-02221 SEAL, OIL',
  'description': '103288-02221 SEAL, OIL',
  'short_description': '<font size="+3">\r\nFits Engines:\r\n</font>NFD13-MEA\r\n NFD13-MEAS\r\n NFD13-MEP\r\n NFD13-MEPA\r\n NFD13-MEPAS\r\n TF110M(E/H/L\r\n TF110N-L\r\n TF120M(E/H/L\r\n TF120ML-XA\r\n TF120N-L\r\n TF80-M(E/H/L\r\n TF90-M(E/H/L',
  'tags': [{'name': '103288-02221'}],
  'sku': 'PX8AKH5JDR',
  'categories': [{'id': 307931}],
  'slug': '103288-02221-seal-oil-yanmar/'}]

我试过的：

一世。 使用 for 循环。

from woocommerce import API

wcapi = API(url=url, consumer_key=consumer_key, consumer_secret=consumer_secret,timeout=50)

for product in my_json_list:
    print(wcapi.post("products", product).json())

这行得通，但要到下个千年才能完成，因为我有数百万个产品页面要创建。

二. concurent.futures模块。

def post_data(product):
    return wcapi.post("products", product).json()

# protect the entry point
if __name__ == '__main__':
    # create the thread pool
    with ThreadPoolExecutor() as ex:
        # issue many asynchronous tasks systematically
        futures = [ex.submit(post_data, page) for page in data]
        # enumerate futures and report results
        for future in futures:
            print(future.result())

当我尝试这个时，我得到一个超时错误，它似乎连接到 API (read timeout=50) ，未来 object 的状态是 pending。

这是完整的错误 output。

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
                       ^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1374, in getresponse
    response.begin()
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1278, in recv_into
    return self.read(nbytes, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1134, in read
    return self._sslobj.read(len, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 451, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 340, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='new.turbo-diesel.co.uk', port=443): Read timed out. (read timeout=50)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/martinhewing/Downloads/Python_Code/Woo_web_pages/create_DEP_pages.py", line 74, in <module>
    futures = [ex.submit(wcapi.post("products", page).json()) for page in data]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/martinhewing/Downloads/Python_Code/Woo_web_pages/create_DEP_pages.py", line 74, in <listcomp>
    futures = [ex.submit(wcapi.post("products", page).json()) for page in data]
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/woocommerce/api.py", line 110, in post
    return self.__request("POST", endpoint, data, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/woocommerce/api.py", line 92, in __request
    return request(
           ^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/requests/adapters.py", line 578, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='new.turbo-diesel.co.uk', port=443): Read timed out. (read timeout=50)

当我查看 ex.summit() function 时，我可以看到状态为待处理。 <Future at 0x124d1f7d0 state=pending>有谁知道这里可能是什么问题？

Answer 1

问题在于服务器可以/可以从一个客户端 IP 获取多少并发连接，因为在现代 web 服务器中，这通常限制为每个客户端的少量连接。

当我尝试使用普通的 for 循环进行 POST 时，我被阻止了。 服务器认为使用并发方法是 DOS 攻击。