Python 3 exception handling and catching

Question

I'm designing a workflow engine for a very specific task and I'm thinking about exception handling.

I've got a main process that calls a few functions. Most of those functions call other more specific functions and so on. There are a few libraries involved so there are a lot of specific errors that can occur. IOError, OSError, AuthenticationException ...

I have to stop the workflow when an error occurs and log it so I can continue from that point when the error is resolved.

Example of what I mean:

def workflow_runner():
    download_file()
    ...
    (more calls with their own exceptions)
    ...


def download_file():
    ftps = open_ftp_connection()
    ftps.get(filename)
    ...
    (more calls with their own exceptions)
    ...


def open_ftp_connection():
    ftps = ftplib.FTP_TLS()
    try:
        ftps.connect(domain, port)
        ftps.login(username, password)
    except ftplib.all_errors as e:
        print(e)
        raise
    return ftps

Your basic, run of the mill, modular functions.

My question is this:

What's considered the best way of doing top to bottom error handling in Python 3?

• To raise every exception to the top and thus put "try except" over each function call up the stack?
• To handle every exception when it happens, log and raise and have no "try except" at the "top"?
• Some better alternative?

Would it be better to just finish and raise the error on the spot or catch it in the "download_file" and/or "workflow_runner" functions?

I ask because if I end up catching everything at the top I feel like I might end up with:

except AError
except BError
...
except A4Error

Answer 1

It depends… You catch an exception at the point where you can do something about it. That differs between different functions and different exception types. A piece of code calls a subsystem (generically speaking any function), and it knows that subsystem may raise exception A , B or C . It now needs to decide what exceptions it expects and/or what it can do about each one of them. In the end it may decide to catch A and B exceptions, but it wouldn't make sense for it to catch C exceptions because it can't do anything about them. This now means this piece of code may raise C exceptions, and its callers need to be aware of that and make the same kinds of decisions.

So different exceptions are caught at different layers, as appropriate.

In more concrete terms, say you have some system which consists of some HTTP object which downloads some stuff from remote servers, some job manager which wrangles a bunch of these HTTP objects and stores their result in a database, and a top level coordinator that starts and stops the job managers. The HTTP objects may obviously raise all sorts of HTTP exceptions when network requests fail, and the job managers may raise exceptions when something's wrong with the database. You will probably let the job managers worry about HTTP errors like 404 , but not about something fundamental like ComputerDoesntHaveANetworkInterface errors; equally DatabaseIsUnreachable exceptions is nothing a job manager can do anything about, and should probably lead to the termination of the application.