Generating rolling averages from a time series, but subselecting based onmonth

Question

I have long time series of weekly data. For a given observation, I want to calculate that week's value versus the average of the three previous years' average value for the same month.

Concrete example: For the 2019-02-15 datapoint, I want to compare it to the value of the average of all the feb-2018, feb-2017, and feb-2016 datapoints.

I want to populate the entire timeseries in this way. (the first three years will be np.nans of course)

I made a really gross single-datapoint example of the calculation I want to do, but I am not sure how to implement this in a vectorized solution. I also am not impressed that I had to use this intermediate helper table "mth_avg".

import pandas as pd
ix = pd.date_range(freq='W-FRI',start="20100101", end='20190301' )
df  = pd.DataFrame({"foo": [x for x in range(len(ix))]}, index=ix) #weekly data
mth_avg = df.resample("M").mean() #data as a monthly average over time
mth_avg['month_hack'] = mth_avg.index.month

#average of previous three years' same-month averages
df['avg_prev_3_year_same-month'] = "?"

#single arbitrary example of my intention
df.loc['2019-02-15', "avg_prev_3_year_same-month"]= (
    mth_avg[mth_avg.month_hack==2]
                    .loc[:'2019-02-15']
                    .iloc[-3:]
                    .loc[:,'foo']
                    .mean() 
                    )


df[-5:]

Answer 1

I think it's actually a nontrivial problem - there's no existing functionality I'm aware of Pandas for this. Making a helper table saves calculation time, in fact I used two. My solution uses a loop (namely a list comprehension) and Pandas datetime awareness to avoid your month_hack . Otherwise I think it was a good start. Would be happy to see something more elegant!

# your code
ix = pd.date_range(freq='W-FRI',start="20100101", end='20190301' )
df  = pd.DataFrame({"foo": [x for x in range(len(ix))]}, index=ix)
mth_avg = df.resample("M").mean()

# use multi-index of month/year with month first
mth_avg.index = [mth_avg.index.month, mth_avg.index.year]
tmp = mth_avg.sort_index().groupby(level=0).rolling(3).foo.mean()
tmp.index = tmp.index.droplevel(0)

# get rolling value from tmp
res = [tmp.xs((i.month, i.year - 1)) for i in df[df.index > '2010-12-31'].index]

# NaNs for 2010
df['avg_prev_3_year_same-month'] = np.NaN
df.loc[df.index > '2010-12-31', 'avg_prev_3_year_same-month'] = res

# output
df.sort_index(ascending=False).head()

            foo     avg_prev_3_year_same-month
2019-03-01  478     375.833333
2019-02-22  477     371.500000
2019-02-15  476     371.500000
2019-02-08  475     371.500000
2019-02-01  474     371.500000