Windows上的SSL：CERTIFICATE_VERIFY_FAILED错误

Question

I'm building a simple program to go through a list of URLs and extract their content using beautiful soup. For the minute I'm just trying to iterate through the list and retrieve the html but I keep getting the following error:

Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "C:\ProgramData\Anaconda3\lib\http\client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\ProgramData\Anaconda3\lib\http\client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\ProgramData\Anaconda3\lib\http\client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\ProgramData\Anaconda3\lib\http\client.py", line 1026, in _send_output
    self.send(msg)
  File "C:\ProgramData\Anaconda3\lib\http\client.py", line 964, in send
    self.connect()
  File "C:\ProgramData\Anaconda3\lib\http\client.py", line 1400, in connect
    server_hostname=server_hostname)
  File "C:\ProgramData\Anaconda3\lib\ssl.py", line 401, in wrap_socket
    _context=self, _session=session)
  File "C:\ProgramData\Anaconda3\lib\ssl.py", line 808, in __init__
    self.do_handshake()
  File "C:\ProgramData\Anaconda3\lib\ssl.py", line 1061, in do_handshake
    self._sslobj.do_handshake()
  File "C:\ProgramData\Anaconda3\lib\ssl.py", line 683, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/thoma/PycharmProjects/fyp/urls_and_prep/parsing_html.py", line 17, in <module>
    response = urllib.request.urlopen(req)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 526, in open
    response = self._open(req, data)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 544, in _open
    '_open', req)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 504, in _call_chain
    result = func(*args)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)>

My program is simple enough but I can't understand or find any good resources for working out what exactly is going on/ how to handle it. I know its something to do with SSL certificates but I'm not sure where how to use them or where to install them etc. I'm just at a bit of a loss with this one as I've never really worked with SSL before. Any guidance or help is greatly appreciated. Here's the code:

import urllib.request
from bs4 import BeautifulSoup

file = open("all_urls.txt", "r")

for line in file:
    print(line)

    try:
        response = urllib.request.urlopen(line)
        html = response.read()
    except ValueError:
        print(ValueError)
        continue
    soup = BeautifulSoup(html, 'lxml')
    print(soup.get_text())

Answer 1

Are you using Windows or Linux? This problem seems to be not on Python, but in Anaconda or in the OS. You can try some simple solutions, like: 1 - Execute the scrypt using other Python installation than the one from Anaconda. 2 - Use the virtualenv to isolate the components of the OS.

Answer 2

Below will fix the problem. But be sure not to use in production as it will be working without verifying the SSL certificate-

import urllib
from bs4 import BeautifulSoup
import ssl

# This is a temporary fix .Be carefule of malicious links
context = ssl._create_unverified_context()
file = open("all_urls.txt", "r")

for line in file:
    print(line)

    try:
        response = urllib.request.urlopen(line, context=context)
        html = response.read()
    except ValueError:
        print(ValueError)
        continue
    soup = BeautifulSoup(html, 'lxml')
    print(soup.get_text())