try except not catching on function?

Question

I am getting this valid error while preprocessing some data:

 9:46:56.323 PM default_model Function execution took 6008 ms, finished with status: 'crash'
 9:46:56.322 PM default_model Traceback (most recent call last):
  File "/user_code/main.py", line 31, in default_model
    train, endog, exog, _, _, rawDf = preprocess(ledger, apps)
  File "/user_code/Wrangling.py", line 73, in preprocess
    raise InsufficientTimespanError(args=(appDf, locDf))

That's occurring here:

async def default_model(request):
    request_json = request.get_json()
    if not request_json:
        return '{"error": "empty body." }'
    if 'transaction_id' in request_json:
        transaction_id = request_json['transaction_id']

        apps = []  # array of apps whose predictions we want, or uempty for all
        if 'apps' in request_json:
            apps = request_json['apps']

        modelUrl = None
        if 'files' in request_json:
            try:
                files = request_json['files']
                modelUrl = getModelFromFiles(files)
            except:
                return package(transaction_id, error="no model to execute")
        else:
            return package(transaction_id, error="no model to execute")

        if 'ledger' in request_json:
            ledger = request_json['ledger']

            try:
                train, endog, exog, _, _, rawDf = preprocess(ledger, apps)
            # ...
            except InsufficientTimespanError as err:
                return package(transaction_id, error=err.message, appDf=err.args[0], locDf=err.args[1])

And preprocess is correctly throwing my custom error:

def preprocess(ledger, apps=[]):
    """
    convert ledger from the server, which comes in as an array of csv entries.
    normalize/resample timeseries, returning dataframes
    """
    appDf, locDf = splitLedger(ledger)

    if len(appDf) < 3 or len(locDf) < 3:
        raise InsufficientDataError(args=(appDf, locDf))

    endog = appDf['app_id'].unique().tolist()
    exog = locDf['location_id'].unique().tolist()

    rawDf = normalize(appDf, locDf)
    trainDf = cutoff(rawDf.copy(), apps)
    rawDf = cutoff(rawDf.copy(), apps, trim=False)

    # TODO - uncomment when on realish data
    if len(trainDf) < 2 * WEEKS:
        raise InsufficientTimespanError(args=(appDf, locDf))

The thing is, it is in a try``except block precisely because I want to trap the error and return a payload with the error, rather than crashing with a 500 error. But its crashing on my custom error, in the try block, anyway. Right on that line calling preprocess .

This must be a failure on my part to conform to proper python code. But I'm not sure what I am doing wrong. The environment is python 3.7

Here's where that error is defined, in Wrangling.py:

class WranglingError(Exception):
    """Base class for other exceptions"""
    pass


class InsufficientDataError(WranglingError):
    """insufficient data to make a prediction"""

    def __init__(self, message='insufficient data to make a prediction', args=None):
        super().__init__(message)
        self.message = message
        self.args = args


class InsufficientTimespanError(WranglingError):
    """insufficient timespan to make a prediction"""

    def __init__(self, message='insufficient timespan to make a prediction', args=None):
        super().__init__(message)
        self.message = message
        self.args = args

And here is how main.py declares (imports) it:

from Wrangling import preprocess, InsufficientDataError, InsufficientTimespanError, DataNotNormal, InappropriateValueToPredict

Answer 1

Your preprocess function is declared async . This means the code in it isn't actually run where you call preprocess , but instead when it is eventually await ed or passed to a main loop (like asyncio.run ). Because the place where it is run is no-longer in the try block in default_model , the exception is not caught.

You could fix this in a few ways:

• make preprocess not async
• make default_model async too, and await on preprocess .

Answer 2

Do the line numbers in the error match up with the line numbers in your code? If not is it possible that you are seeing the error from a version of the code before you added the try...except?