Pandas: ignore year in index when plotting but keep hour/day/month

Question

I have the following data series:

                     values
Date                              
2013-01-01 00:00:00            NaN
2013-01-01 01:00:00       0.041702
2013-01-01 02:00:00       0.042505
2013-01-01 03:00:00       0.030535
...
2020-12-30 21:00:00       0.525059
2020-12-30 22:00:00       0.249274
2020-12-30 23:00:00       0.024965     

I want to:

1. roll all years,
2. align them by day of the week,
3. calculate statistics such as the mean for that day across the eight years of data---Yes, eight points is too few for statistics. Eg, point 0 would be the mean of the values of the first Tuesdays of each year at 00:00:00, or 2013-01-01 00:00:00 , 2014-01-07 00:00:00 , 2015-01-06 00:00:00 , etc.

Plotting this would basically result in a plot with a single line based on about 365 point estimates (ignoring leap years and days with fewer data at the end of the year). I tried starting with pivot tables as suggested here but failed miserably:

df_pv = pd.pivot_table(series.to_frame(), columns=series.index.year)

Exception has occurred: AttributeError 'Series' object has no attribute 'columns'

Any ideas?

Answer 1

It's exactly as you have written down, just a groupby() not a pivot.

import datetime as dt
import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=[10,6])

d = pd.date_range(dt.date(2013,1,1), dt.date(2021,1,1), freq="H")
df = pd.DataFrame({"Date":d,"values":np.random.uniform(0,1,len(d))})


l = df.groupby([df.Date.dt.dayofweek,df.Date.dt.isocalendar().week,df.Date.dt.time]).agg({"values":"mean"}).plot(ax=ax)
l = ax.set_xticklabels(ax.get_xticklabels(), rotation = 90)