Creating a new column by shifting an existing column to 1 row down using pandas

Question

I am working on sport. The purpose is to record current eventdatetime and PreviousEventTime in a game. I have a sample dataset in the below link.

https://drive.google.com/open?id=1DUNrWPFwrkZHpq_KeA4rZCJ94sbpUEDI

In this file, there are 11 columns. the event are collected based on time. For this re-arrange, i will be using the following columns gsm_ID , eventdatetime columns

I want to create a new column PreviousEventTime that take n-1 row of the eventdatetime column. That means for every gsm_ID , there will be the first eventdatetime . The new column will represent the next event time as compared to the time column.

gsm_ID eventdatetime PreviousEventTime

2462794 08/11/2017 18:46 08/11/2017 18:45

2462794 08/11/2017 18:49 08/11/2017 18:46

2462794 08/11/2017 19:13 08/11/2017 18:49

2462794 08/11/2017 19:31 08/11/2017 19:13

2462794 08/11/2017 20:09 08/11/2017 19:31

2462795 08/12/2017 17:39 08/12/2017 16:30

2462795 08/12/2017 17:44 08/12/2017 17:39

Above example is just for two games. You can differentiate by gsm_id . The for row at PreviousEventTime will always be matchdatetime. I will have 100 over games. but the process will repeat as above-mentioned example.

eventdata ['PreviousEventTime-1'] = eventdata.groupby(['gsm_id'])['eventdatetime'].shift(-1)

But it only works for the first gsm_ID . It did not work for the other gsm_ID . The output from above script is below:

Your advice would be much appreciated. Regards, zephyr

Answer 1

Sorting properly solved the problem. I added in the following sorting and indexing:

eventdata = eventdata.set_index(['gsm_id']) .sort_index(ascending =True)

eventdata=eventdata.sort_values(['matchdatetime','time'],ascending=[True,True])

eventdata ['PreviousEventTime-1'] = eventdata.groupby(['gsm_id','matchdatetime'])['eventdatetime'].shift(1, axis = 0)

But the remaining part is to fill NaT by matchdatetime . Thanks everyone for advising me. Regards zephyr