Subtracting between rows of different columns in an indexed dataframe in python

Question

I have an indexed dataframe (indexed by type then date) and would like to carry out a subtraction between the end time of the top row and start time of the next row in hours :

type    date             start_time                  end_time       code
A      01/01/2018         01/01/2018 9:00       01/01/2018 14:00      525
       01/02/2018         01/02/2018 5:00       01/02/2018 17:00      524
       01/04/2018         01/04/2018 8:00       01/04/2018 10:00      528
B      01/01/2018         01/01/2018 5:00       01/01/2018 14:00      525
       01/04/2018         01/04/2018 2:00       01/04/2018 17:00      524
       01/05/2018         01/05/2018 7:00       01/05/2018 10:00      528

I would like to get the resulting table with a new column['interval']:

type    date             interval
A      01/01/2018           -
       01/02/2018           15
       01/04/2018           39
B      01/01/2018           -
       01/04/2018           60
       01/05/2018           14

The interval column is in hours

Answer 1

You can convert start_time and end_time to datetime format, then use apply to subtract the end_time of the previous row in each group (using groupby ). To convert to hours, divide by pd.Timedelta('1 hour') :

df['start_time'] = pd.to_datetime(df['start_time'])
df['end_time'] = pd.to_datetime(df['end_time'])

df['interval'] = (df.groupby(level=0,sort=False).apply(lambda x: x.start_time-x.end_time.shift(1)) / pd.Timedelta('1 hour')).values

>>> df
                         start_time            end_time    code    interval
type date                                                              
A    01/01/2018  2018-01-01 09:00:00  2018-01-01 14:00:00   525       NaN
     01/02/2018  2018-01-02 05:00:00  2018-01-02 17:00:00   524      15.0
     01/04/2018  2018-01-04 08:00:00  2018-01-04 10:00:00   528      39.0
B    01/01/2018  2018-01-01 05:00:00  2018-01-01 14:00:00   525       NaN
     01/04/2018  2018-01-04 02:00:00  2018-01-04 17:00:00   524      60.0
     01/05/2018  2018-01-05 07:00:00  2018-01-05 10:00:00   528      14.0