catching warnings from numpy and scipy with try except

Question

I'm fitting an ExponentialSmoothing model from statsmodels with version 0.10.1 (statsmodels.tsa.holtwinters.ExponentialSmoothing) - I have to do this repeatedly on groups in a pandas groupby object, but the point is I'm looping through datasets to fit a new model on each group of data.

In some iterations of the loop, but not all, statsmodels raises a ConvergenceWarning, or a RuntimeWarning, and I want to capture these warnings and record them in the result dataframe next to the prediction.

I've tried using a warnings context manager so that I can raise warnings like they're exceptions, and added try/except blocks

The problem with this approach is that if one of these warnings gets caught, the model isn't actually fit because the try block gets skipped and, worst of all, I end up predicting using a model fit on a previous iteration.

import pandas as pd
from statsmodels.tsa.holtwinters import ExponentialSmoothing, HoltWintersResults
from statsmodels.tools.sm_exceptions import ConvergenceWarning
import numpy as np
from loguru import logger

concat_region_keys = []
forecast_result_dfs = []
region_df_groups = deseasonalized_search_counts \
    .sort_index() \
    .groupby('region_id') \
    ['region_deseasonalized_observations']
with warnings.catch_warnings():
    warnings.simplefilter("error")
    for i, (region_id, df_region_group) in enumerate(region7_df_groups):
        concat_region_keys.append(region_id)
        region_guard_rail = df_region_group.iloc[-1]  # most recent observation
        err_msg = np.nan
        try:
            holt_winters_result_object = ExponentialSmoothing(endog=df_region_group).fit()
        except ConvergenceWarning as ce:
            logger.warning(f"{i} region {region_id}: ConvergenceWarning {ce}")
            err_msg = f"ConvergenceWarning: {ce}"
        except RuntimeWarning as re:
            logger.warning(f"{i} region {region_id}: RuntimeWarning {re}")
            err_msg = f"RuntimeWarning: {re}"
        forecast_result = holt_winters_result_object \
            .forecast(periods_ahead) \
            .assign(
                fit_call_warning=err_msg,
            )
        forecast_result_dfs.append(forecast_result)

What's the correct way, with try/except or some other python construct, to do the following for each iteration of the loop?

1. fit the model
2. capture any warnings that were raised so I can record them in a variable or dataframe.

Answer 1

One issue is this

"In some iterations of the loop, but not all, statsmodels raises a ConvergenceWarning, or a RuntimeWarning, and I want to capture these warnings and record them in the result dataframe next to the prediction."

You can't capture a prediction that threw an error the error was the output of the function. You can save the predictions up to the point an error occurred. You need to decided what you want to do with your model after it throws an error like that. You could use continue like @ALollz said or you could move on to the next group you want to fit but that is a decision you need to make. It's a design choice wether to abandon the set that threw an error while fitting or decide to move on, we can't decide that for you.

Answer 2

Check this website https://www.programcreek.com/python/example/369/warnings.filterwarnings

Use the with statement

with warnings.catch_warnings():