HTTPS 中的 Flask 代理服务器错误 - 适用于 HTTP 但不适用于 HTTPS

Question

我们在烧瓶上设置了一个基本的代理服务器。 它在HTTP中运行良好，但是当使用HTTPS发出请求时，它会引发一些难以调试的问题。

在HTTPS中，flask 似乎没有正确解析请求。

更详细的描述如下

服务器代码

from flask import Flask, request
import requests
from requests import exceptions


application = Flask(__name__)

@application.route("/",methods=["GET","POST","DELETE","PATCH","PUT"])
def route():
    try:
        response = requests.request(method=request.method, url=request.url, headers=request.headers,timeout=60)
        print(response.status_code, response.text)
        return (response.content, response.status_code, response.headers.items())
    except exceptions.ProxyError:
        return (b"Error", 408, request.headers.items())

if __name__ == "__main__":
    application.run(host="0.0.0.0", port=39100, debug=True, use_reloader=False,ssl_context=("certs/cert.pem","certs/key.pem"))

索取示例代码

import requests

proxies = {"http":"https://127.0.0.1:39100", "https":"https://127.0.0.1:39100"}

# does not work if the request is http://ipv4.icanhazip.com/
# works if the request is http://ipv4.icanhazip.com/
testing = requests.get(url="https://ipv4.icanhazip.com/",proxies=proxies,verify=False) 
print(testing)
print(testing.content)

错误

/Users/a/PycharmProjects/invisiProxy/venv/bin/python /Users/a/PycharmProjects/invisiProxy/sudoRequest.py
Traceback (most recent call last):
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 700, in urlopen
    self._prepare_proxy(conn)
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 994, in _prepare_proxy
    conn.connect()
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/urllib3/connection.py", line 369, in connect
    self._tunnel()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 924, in _tunnel
    raise OSError(f"Tunnel connection failed: {code} {message.strip()}")
OSError: Tunnel connection failed: 404 NOT FOUND

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ipv4.icanhazip.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 404 NOT FOUND')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/a/PycharmProjects/invisiProxy/sudoRequest.py", line 6, in <module>
    testing = requests.get(url="https://ipv4.icanhazip.com/",proxies=proxies,verify=False)
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/adapters.py", line 559, in send
    raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='ipv4.icanhazip.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 404 NOT FOUND')))

Process finished with exit code 1

Answer 1

对 HTTPS requests使用proxies会导致客户端通过CONNECT方法使用 HTTP 隧道。 客户端必须有到目标的直接 TCP 通道才能在没有任何 MITM 代理的情况下建立端到端加密（请参阅urllib3描述以了解这种情况 - https://urllib3.readthedocs.io/en/stable/ advanced-usage.html#http-and-https-proxies ）。

您的烧瓶服务器不支持 CONNECT 方法，因此它在请求中返回 404。 可以在提供的 MVP 服务器上观察到该行为：

# client request http://ipv4.icanhazip.com/
127.0.0.1 - - [24/Jun/2022 12:08:12] "GET http://ipv4.icanhazip.com/ HTTP/1.1" 200 -

# client request https://ipv4.icanhazip.com/
127.0.0.1 - - [24/Jun/2022 12:03:48] "CONNECT ipv4.icanhazip.com:443 HTTP/1.0" 404

从技术上讲，可以执行预期的行为......

# openssl s_client --connect 127.0.0.1:39100
CONNECTED(00000003)
---
Certificate chain
 0 s:C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = xxxx
   i:C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = xxxx
---
read R BLOCK
GET https://ipv4.icanhazip.com/ HTTP/1.1
Host: ipv4.icanhazip.com

HTTP/1.1 200 OK
Server: Werkzeug/2.1.2 Python/3.7.3
...

IP_REDACTED
read:errno=0

...并且这种情况在urllib3级别（由实际连接requests使用）记录和“调整”，使用 https://urllib3.readthedocs.io/en/stable/reference/urllib3.poolmanager.html#urllib3 的use_forwarding_for_https 。代理管理器

但

• 一般不推荐这种用法，因为它违反了 TLS 本身的设计（客户端和资源之间的端到端加密）。

• 在当前的requests实现中，似乎没有办法通过标准 api 从开发人员那里传递所需的标志。 get_connection函数不允许将任何额外的proxy_kwargs传递给urllib3.ProxyManager

https://github.com/psf/requests/blob/da9996fe4dc63356e9467d0a5e10df3d89a8528e/requests/adapters.py#L352

    def get_connection(self, url, proxies=None):
        ...
        if proxies:
            ...
            proxy_manager = self.proxy_manager_for(proxy)
            conn = proxy_manager.connection_from_url(url)


https://github.com/psf/requests/blob/da9996fe4dc63356e9467d0a5e10df3d89a8528e/requests/adapters.py#L201
    def proxy_manager_for(self, proxy, **proxy_kwargs):
        ...
        manager = self.proxy_manager[proxy] = proxy_from_url(
                ...
                **proxy_kwargs,
            )

我建议使用完整的 http 代理而不是烧瓶服务器。