Python 3 Requests errors

Question

I'm working through Python Crash Course 2nd Ed. and in the text is some code for accessing APIs. My code is copied from the text and is as follows:

import requests
import json
from operator import itemgetter


#Fetch top stories and store in variable r
url = 'https://hacker-news.firebaseio.com/v0/topstories.json'
r = requests.get(url)
print(f"Status code: {r.status_code}")

# #Explore data structure
# response_dict = r.json()
# readable_file = 'hn_readable.json'
# with open(readable_file, 'w') as f:
#   json.dump(response_dict, f, indent=4)

submission_ids = r.json()
submission_dicts = []
for submission_id in submission_ids[:30]:
    #Make API call for each article
    url = f"https://hacker-news.firebasio.com/v0/item/{submission_id}.json"
    r = requests.get(url)
    print(f"id: {submission_id}\tstatus code: {r.status_code}")
    response_dict = r.json()

    #Store dictionary of each article
    submission_dict = {
        'title': response_dict['title'],
        'score': response_dict['score'],
        'comments': response_dict['descendants'],
        'link': response_dict['url'],
    }
    submission_dicts.append(submission_dict)
    

#Sort article by score
submission_dicts = sorted(submission_dicts, key=itemgetter('score'), reverse = True)

#Display information about each article, ranked by score
for submission_dict in submission_dicts:
    print(f"Article title: {submission_dict['title']}")
    print(f"Article link: {submission_dict['url']}")
    print(f"Score: {submission_dict['score']}")

However, this is now returning the following error messages:

Status code: 200
Traceback (most recent call last):
  File "C:\Users\snack\AppData\Roaming\Python\Python37\site-packages\urllib3\connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "C:\Users\snack\AppData\Roaming\Python\Python37\site-packages\urllib3\connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "C:\Users\snack\AppData\Roaming\Python\Python37\site-packages\urllib3\connectionpool.py", line 976, in _validate_conn
    conn.connect()
  File "C:\Users\snack\AppData\Roaming\Python\Python37\site-packages\urllib3\connection.py", line 370, in connect
    ssl_context=context,
  File "C:\Users\snack\AppData\Roaming\Python\Python37\site-packages\urllib3\util\ssl_.py", line 377, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "C:\Users\snack\Python\lib\ssl.py", line 423, in wrap_socket
    session=session
  File "C:\Users\snack\Python\lib\ssl.py", line 870, in _create
    self.do_handshake()
  File "C:\Users\snack\Python\lib\ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1076)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\snack\AppData\Roaming\Python\Python37\site-packages\requests\adapters.py", line 449, in send
    timeout=timeout
  File "C:\Users\snack\AppData\Roaming\Python\Python37\site-packages\urllib3\connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "C:\Users\snack\AppData\Roaming\Python\Python37\site-packages\urllib3\util\retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='hacker-news.firebasio.com', port=443): Max retries exceeded with url: /v0/item/23273247.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1076)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\snack\Python\proj_2\hn_submissions.py", line 24, in <module>
    r = requests.get(url)
  File "C:\Users\snack\AppData\Roaming\Python\Python37\site-packages\requests\api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\snack\AppData\Roaming\Python\Python37\site-packages\requests\api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\snack\AppData\Roaming\Python\Python37\site-packages\requests\sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\snack\AppData\Roaming\Python\Python37\site-packages\requests\sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\snack\AppData\Roaming\Python\Python37\site-packages\requests\adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='hacker-news.firebasio.com', port=443): Max retries exceeded with url: /v0/item/23273247.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1076)')))
[Finished in 3.6s]

I have almost no experience with this, but from what I can tell, some authentication is failing and not letting my program access the API, but I have no idea why. I've tried limiting the number of API calls by removing the loop, but it doesn't seem to help. I also tried adding the verify=False parameter into the requests.get lines, but that just kicked up different errors.

Answer 1

There is nothing wrong with the API call itself.

As you visit the site https://hacker-news.firebaseio.com/v0/topstories.json you can see the expected list in the browser. (Your first and working api call)

As the first number in this list is 23277594, the script start with this request https://hacker-news.firebasio.com/v0/item/23277594.json , but visiting this url via the browser will also result in warnings. (your second and failing api call)

Answer 2

Alright, it was typos (of course). The url in my code was https...firebasio....json instead of https...firebas e io....json. One of the results is still not working, but I'm assuming that's due to the article not having comments (ie descendants), so some try/ except should fix that.