由于Python中的请求的网址不安全而捕获SSLError吗？

Question

I have a list of a few thousand URLs and noticed one of them is throwing as SSLError when passed into requests.get() . Below is my attempt to work around this using both a solution suggested in this similar question as well as a failed attempt to catch the error with a "try & except" block using ssl.SSLError :

url = 'https://archyworldys.com/lidl-recalls-puff-pastry/'

session = requests.Session()
retry = Retry(connect=3, backoff_factor=0.5)
adapter = HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://', adapter)

try:
    response = session.get(url,allow_redirects=False,verify=True)
except ssl.SSLError:
    pass

The error returned at the very end is:

SSLError: HTTPSConnectionPool(host='archyworldys.com', port=443): Max retries exceeded with url: /lidl-recalls-puff-pastry/ (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),))

When I opened the URL in Chrome, I get a "Not Secure" / "Privacy Error" that blocks the webpage. However, if I try the URL with HTTP instead of HTTPS (eg ' http://archyworldys.com/lidl-recalls-puff-pastry/ ') it works just fine in my browser. Per this question , setting verify to False solves the problem, but I prefer to find a more secure work-around.

While I understand a simple solution would be to remove the URL from my data, I'm trying to find a solution that let's me proceed (eg if in a for loop ) by simply skipping this bad URL and moving on the next one.

Answer 1

The error I get when running your code is:

requests.exceptions.SSLError:
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

Based on this one needs to catch requests.exceptions.SSLError and not ssl.SSLError , ie:

try:
    response = session.get(url,allow_redirects=False,verify=True)
except requests.exceptions.SSLError:
    pass

While it looks like the error you get is different this is probably due the code you show being not exactly the code you are running. Anyway, look at the exact error message you get and figure out from this which exception exactly to catch. You might also try to catch a more general exception like this and by doing this get the exact Exception class you need to catch:

try:
    response = session.get(url,allow_redirects=False,verify=True)
except Exception as x:
    print(type(x),x)
    pass