handling exceptions from python requests

Question

My function is executed with threads:

def getdata(self, page, ...):
    tries = 10
    for n in range(tries):
        try:
            ...
            datarALL = []
            url = 'http://website/...'.format(...)
            responsedata = requests.get(url, data=data, headers=self.hed, verify=False)
            responsedata.raise_for_status()
            if responsedata.status_code == 200:  # 200 for successful call
                ...
                    if ...
                        break   
        except (ChunkedEncodingError, requests.exceptions.HTTPError) as e:
            print ("page #{0} run #{1} failed. Returned status code {2}. Reason: {3}. Msg: {4}. Retry.".format(page, n, responsedata.status_code, responsedata.reason, sys.exc_info()[0]))
            if n == tries - 1:
                print ("page {0} could not be imported. Max retried reached.".format(page))
                os._exit(1)  #One thread max retried - close all threads and 
    return datarALL

as follows:

    with ThreadPoolExecutor(max_workers=num_of_workers) as executor:
        futh = [(executor.submit(self.getdata, page,...)) for page in pages]
        for data in as_completed(futh):
            datarALL.extend(data.result())
    print ("Finished generateing data.")
    return datarALL

Sometimes I get unexpected exceptions like: ConnectionResetError: [Errno 104] Connection reset by peer which shuts down my program. I want to change my code so that no matter which exception happens the thread will allways retry untill the if n == tries - 1: is met. I don't want my thread to shut down due to random exception.

I read the request exceptions info page but I don't see how I can catch all the exceptions without manually listing all of them. Is there a generic way to do this?

Basicly I want something like:

        except (ALL EXCEPTIONS from Requests) as e:
            print ("page #{0} run #{1} failed. Returned status code {2}. Reason: {3}. Msg: {4}. Retry.".format(page, n, responsedata.status_code, responsedata.reason, sys.exc_info()[0]))
            if n == tries - 1:
                print ("page {0} could not be imported. Max retried reached.".format(page))
                os._exit(1)  #One thread max retried - close all threads and 
    return datarALL

How can I do that?

EDIT: using

    except Exception as e:
        print ("page #{0} run #{1} failed. Returned status code {2}. Reason: {3}. Msg: {4}. Retry.".format(page, n, responsedata.status_code, responsedata.reason, sys.exc_info()[0]))
        if n == tries - 1:
            print ("page {0} could not be imported. Max retried reached.".format(page))
            os._exit(1)  #One thread max retried - close all threads and 
return datarALL

Does not catch. it gives me this:

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.5/site-packages/urllib3/response.py", line 331, in _error_catcher
    yield
  File "/home/ubuntu/.local/lib/python3.5/site-packages/urllib3/response.py", line 640, in read_chunked
    chunk = self._handle_chunk(amt)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/urllib3/response.py", line 595, in _handle_chunk
    returned_chunk = self._fp._safe_read(self.chunk_left)
  File "/usr/lib/python3.5/http/client.py", line 607, in _safe_read
    chunk = self.fp.read(min(amt, MAXAMOUNT))
  File "/usr/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

The loop does not retry. The run is terminated...

EDIT2:

    except requests.exceptions.RequestException as e:
        print ("page #{0} run #{1} failed. Returned status code {2}. Reason: {3}. Msg: {4}. Retry.".format(page, n, responsedata.status_code, responsedata.reason, sys.exc_info()[0]))
        if n == tries - 1:
            print ("page {0} could not be imported. Max retried reached.".format(page))
            os._exit(1)  #One thread max retried - close all threads and 
return datarALL

Also does not catch the ConnectionResetError: [Errno 104] Connection reset by peer listed above.

Answer 1

Catching all exceptions is usually considered a bad practice since it could hide some problems.

That said, Python exception benefit from inheritance and catching a base exception will catch every exception inherited from this base exception.

See the Python standard exception hierarchy for detail.

You can see that the root exception is BaseException , but this one should never be catched since it will catch Ctrl+C interruptions and generator exits. If you want to catch every exceptions types, you could catch Exception .

You could also want to catch only exception from requests . In that case, according to the doc , it can be done by catching the base exception of the requests module: RequestException

---

If you want to catch both requests exceptions and ConnectionResetError (which is a Python standard exception), you have to specify both in your except clause:

except (requests.exceptions.RequestException,
        ConnectionResetError) as err:
    # some code

Or if you want to be less specific and catch all possible connection errors you can use ConnectionError instead of ConnectionResetError . (see the exceptions hierarchy )

Finally, you may want to react differently to each exception type. In this case, you can do this:

try:
    # something
except ConnectionError as err:
    # manage connection errors
except requests.exceptions.RequestException as err:
    # manage requests errors

Answer 2

检查您的settings.py - 是否在已安装的应用程序中添加了频道。