Python: requests.exceptions.ConnectionError: HTTPSConnectionPool

Question

I am trying to monitor some application URL via below code but it's giving me error. Please suggest what is the issue in here?

Code:

#!/usr/bin/env python3

import requests
import os

f = open("sites.txt", 'r')
dns = f.readlines()
for obj in dns:
#    try:
        url = "https://" + obj
        print(obj)
        response = requests.get(url, verify=False, timeout=5)
        if response.status_code != 200:
            print("Site not rechable", url)

[user@server]$ cat sites.txt
www.google.com
web1.com

result:

[user@server]$ python3 monitor.py
www.google.com

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 157, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 300, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f3cf8d221d0>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.google.com%0a', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f3cf8d221d0>: Failed to establish a new connection: [Errno -2] Name or service not known',))

I can confirm that google is reachable from the server and I am able to get the curl result.

Answer 1

Simply, the second website is dead https://web1.com/ , so requests is throwing an exception. You need to intercept it: You need to check it:

import requests
dns = ['www.google.com', 'web1.com']
for obj in dns:
        url = "https://" + obj
        try:
            response = requests.get(url, verify=False, timeout=5)
        except requests.exceptions.ConnectionError:
            print("Site not rechable", url)