ARMA forecasting, problems with trend elimination

Question

I want to do a rough ARMA forecasting for it, ie to get knowledge more about how to use stats model library and see how it works. So firstly I launched the example that is somewhere in the web, but ARMA fitting and prediction does not work, as MLE does not converge. I decided that series is not stationary, so firstly, I want to eliminate trend, and this is a challenge to me. Here is the code:

   import pandas.io.data as web
import statsmodels.api as sm
import statsmodels.tsa.api as tsa
import datetime
import statsmodels.formula.api as smf


start = datetime.datetime(2010, 1, 1)
end = datetime.datetime(2010, 1, 27)
f = web.DataReader("F", 'yahoo', start, end)
print f

#+++++++++++++++TREND+++++++++++++++
atrend = tsa.add_trend(f['Close'].values,trend='ctt')
print atrend
#+++++++++++++++fitting ARMA++++++++

arma =tsa.ARMA(f['Close'].values, order =(2,2))
results= arma.fit()

So, printing the 'atrend' gives me:

[[  10.28    1.      1.      1.  ]
 [  10.96    1.      2.      4.  ]
 [  11.37    1.      3.      9.  ]
 [  11.66    1.      4.     16.  ]
 [  11.69    1.      5.     25.  ]
 [  12.11    1.      6.     36.  ]
 [  11.87    1.      7.     49.  ]
 [  11.68    1.      8.     64.  ]
 [  11.76    1.      9.     81.  ]
 [  11.6     1.     10.    100.  ]
 [  11.75    1.     11.    121.  ]
 [  11.51    1.     12.    144.  ]
 [  11.18    1.     13.    169.  ]
 [  10.52    1.     14.    196.  ]
 [  11.03    1.     15.    225.  ]
 [  11.19    1.     16.    256.  ]
 [  11.55    1.     17.    289.  ]]

Which I do not understand completely. I asked to have a ctt trend which is : at^2+bt+c I calculated, for comparison, both at^2+bt+c, at^3+bt^2+ct+d in excel, and got the following values.

10,494523
10,780752
11,031687
11,247328
11,427675
11,572728
11,682487
11,756952
11,796123
11,8
11,768583
11,701872
11,599867
11,462568
11,289975
11,082088
10,838907

for parabola, y = -1.7647x2 + 33.917x + 1017.3 ,and:

10,00432
10,65848
11,1547
11,5105
11,7434
11,87092
11,91058
11,8799
11,7964
11,6776
11,54102
11,40418
11,2846
11,1998
11,1673
11,20462
11,32928

for cubic equation y = 0.292x3 - 9.649x2 + 92.319x + 917.47. Even now, I do not have an idea how to insert these values to the arma =tsa.ARMA( f['Close'].values , order =(2,2)) in order to check if I can do the next task.

All in all my idea is to make a brief presentation how easily ARMA forecasting can be used in python, however for me it is not the case.

Answer 1

It's supposed to be easy but handling of explanatory variables, exog, and trend still have some tricky parts. ( exog handling was a late addition to ARMA.)

ARMA has an exog keyword for additional explanatory variables for the mean prediction. The constant is included via the trend keyword of the fit method and currently cannot be included in exog . Unfortunately, the trend keyword doesn't allow for arbitrary order trends.

So in your case something like this should work:

arma = tsa.ARMA(f['Close'].values, exog=f[['trend1', 'trend2']] order=(2,2))

You will need the same kind of trend exog for out of sample prediction in predict .

Here https://gist.github.com/josef-pkt/1ea164439b239b228557 is a secret gist of mine that I used to figure out how to use exog in ARMA.

Current master and upcoming statsmodels 0.7 fix some timing problems when we have exog for out of sample prediction.

In general, it is possible to regress the time series on the explanatory variables with OLS or another method, and then use the residuals to estimate an ARMA model. This works for forecasting, however the standard errors of the ARMA estimate and the prediction standard errors will be wrong because they don't take the initial estimate into account (AFAIK).