MemoryError: Unable to allocate 797. MiB for an array with shape (51, 51, 40169) and data type float64

Question

sarima_preds = []
history = [x for x in train_series]

for t in range(len(test_series)):
model = SARIMAX(history, order=(1,1,1), seasonal_order=(1,1,1,24), enforce_stationarity=False, enfore_invertibility=False)
model_fit = model.fit()
output = model_fit.forecast()
sarima_preds.append(output[0])
history.append(test_series[t])

sarima_pred_series = pd.Series(sarima_preds, index=test_series.index)


MemoryError                               Traceback (most recent call last)
File statsmodels\tsa\statespace\_kalman_smoother.pyx:1014, in statsmodels.tsa.statespace._kalman_smoother.dKalmanSmoother.allocate_arrays()

I'm getting memory error after it runs for approximately 1-2 hours. I'm unable to understand what shall I do next. I tried searching for sources and I'm unable to get any one of them.

Answer 1

You could try either pre-allocating the memory instead of appending to lists or threshold which results are saved if you don't care about having each one. To pre-allocate...

samrima_preds = np.empty(len(test_series))

then when ready...

sarima_preds[t] = output[0]

Alternatively, change the dtype, to lower the precision eg fp32 if this is applicable.