计算pandas中15分钟数据的每日差异

Question

I have a huge dataframe of open and close prices recorded every 15 minutes of the day. The day starts at 9:45 and ends at 16:15. My current df looks like this:

                      open_p  close_p
date                                
2013-12-20 09:45:00   -1.14    -1.12
2013-12-20 10:00:00   -1.12    -1.12
2013-12-20 10:15:00   -1.12    -1.11
2013-12-20 10:30:00   -1.11    -1.10
...
2013-12-20 15:30:00   -1.13    -1.14
2013-12-20 15:45:00   -1.14    -1.14
2013-12-20 16:00:00   -1.13    -1.06
2013-12-20 16:15:00   -1.05    -1.01
2013-12-23 09:45:00   -1.02    -1.02
2013-12-23 10:00:00   -1.02    -1.02
2013-12-23 10:15:00   -1.03    -1.07
2013-12-23 10:30:00   -1.06    -1.08
....
2013-12-23 15:30:00   -1.11    -1.14
2013-12-23 15:45:00   -1.13    -1.12
2013-12-23 16:00:00   -1.12    -1.09
2013-12-23 16:15:00   -1.09    -1.13
...

I would like to calculate difference between close_p at 16:15 and open_p at 9:45 for each day. For example daily change column for 2013-12-20 equals -1.01 - (-1.14). The results should look like this:

                      open_p  close_p  daily_change
date                                
2013-12-20 09:45:00   -1.14    -1.12     0.13
2013-12-20 10:00:00   -1.12    -1.12     0.13
2013-12-20 10:15:00   -1.12    -1.11     0.13
2013-12-20 10:30:00   -1.11    -1.10     0.13
...
2013-12-20 15:30:00   -1.13    -1.14     0.13
2013-12-20 15:45:00   -1.14    -1.14     0.13
2013-12-20 16:00:00   -1.13    -1.06     0.13
2013-12-20 16:15:00   -1.05    -1.01     0.13
2013-12-23 09:45:00   -1.02    -1.02    -0,11
2013-12-23 10:00:00   -1.02    -1.02    -0,11
2013-12-23 10:15:00   -1.03    -1.07    -0,11
2013-12-23 10:30:00   -1.06    -1.08    -0,11
....
2013-12-23 15:30:00   -1.11    -1.14    -0,11
2013-12-23 15:45:00   -1.13    -1.12    -0,11
2013-12-23 16:00:00   -1.12    -1.09    -0,11
2013-12-23 16:15:00   -1.09    -1.13    -0,11

What's the fastest and most convenient way of getting this done?

Answer 1

You can groupby on date, agg on first and last, then find the difference:

print (df.groupby(pd.Grouper(freq="D"))
         .agg({"open_p":"first", "close_p":"last"})
         .diff(axis=1)["close_p"])

date
2013-12-20    0.13
2013-12-21     NaN
2013-12-22     NaN
2013-12-23   -0.11
Freq: D, Name: close_p, dtype: float64

Answer 2

Use GroupBy.transform with GroupBy.last and GroupBy.first values and subtract to new column:

g = df.groupby(pd.Grouper(freq='d'))
df['daily_change'] = g['close_p'].transform('last').sub(g['open_p'].transform('first'))
print (df)
                     open_p  close_p  daily_change
date                                              
2013-12-20 09:45:00   -1.14    -1.12          0.13
2013-12-20 10:00:00   -1.12    -1.12          0.13
2013-12-20 10:15:00   -1.12    -1.11          0.13
2013-12-20 10:30:00   -1.11    -1.10          0.13
2013-12-20 15:30:00   -1.13    -1.14          0.13
2013-12-20 15:45:00   -1.14    -1.14          0.13
2013-12-20 16:00:00   -1.13    -1.06          0.13
2013-12-20 16:15:00   -1.05    -1.01          0.13
2013-12-23 09:45:00   -1.02    -1.02         -0.11
2013-12-23 10:00:00   -1.02    -1.02         -0.11
2013-12-23 10:15:00   -1.03    -1.07         -0.11
2013-12-23 10:30:00   -1.06    -1.08         -0.11
2013-12-23 15:30:00   -1.11    -1.14         -0.11
2013-12-23 15:45:00   -1.13    -1.12         -0.11
2013-12-23 16:00:00   -1.12    -1.09         -0.11
2013-12-23 16:15:00   -1.09    -1.13         -0.11

Another idea is use Series.at_time , remove times converting DatetimeIndex to dates and last Series.map :

f = lambda x: x.date()
s = (df['close_p'].at_time('16:15:00').rename(f)
       .sub(df.at_time('09:45:00').rename(f)['open_p']))

df['daily_change'] = df.index.to_frame()['date'].dt.date.map(s)

print (df)
                     open_p  close_p  daily_change
date                                              
2013-12-20 09:45:00   -1.14    -1.12          0.13
2013-12-20 10:00:00   -1.12    -1.12          0.13
2013-12-20 10:15:00   -1.12    -1.11          0.13
2013-12-20 10:30:00   -1.11    -1.10          0.13
2013-12-20 15:30:00   -1.13    -1.14          0.13
2013-12-20 15:45:00   -1.14    -1.14          0.13
2013-12-20 16:00:00   -1.13    -1.06          0.13
2013-12-20 16:15:00   -1.05    -1.01          0.13
2013-12-23 09:45:00   -1.02    -1.02         -0.11
2013-12-23 10:00:00   -1.02    -1.02         -0.11
2013-12-23 10:15:00   -1.03    -1.07         -0.11
2013-12-23 10:30:00   -1.06    -1.08         -0.11
2013-12-23 15:30:00   -1.11    -1.14         -0.11
2013-12-23 15:45:00   -1.13    -1.12         -0.11
2013-12-23 16:00:00   -1.12    -1.09         -0.11
2013-12-23 16:15:00   -1.09    -1.13         -0.11