提取特定时间段内时间戳列中的最大时间间隔

Question

I believe that my problem is really straightforward and there must be a really easy way to solve this issue, however as I am quite new with Python, specially pandas, I could not sort it out by my own.

I made up the following dataframe, which represents a way simpler scenario of what I have been working on. I am looking for a way in which I can gather the maximum timestamp interval between each index for each 10 minutes. I am designing a filter and hence I want to eliminate to able to visualize the maximum time difference for each 10 minutes.

            Timestamp      Category  ...       Class           Speed
0     2013-08-14 22:00:00         1  ...          1               1
1     2013-08-14 22:00:01         1  ...          2               1
2     2013-08-14 22:00:05         1  ...          0               1.1
3     2013-08-14 22:00:07         1  ...          1               1.2
4     2013-08-14 22:00:14         1  ...          3               1.2
5     2013-08-14 22:00:15         1  ...          0               1.2
6     2013-08-14 22:00:16         1  ...          1               1.2
7     2013-08-14 22:00:27         1  ...          2               1.2
8     2013-08-14 22:00:38         1  ...          1               1.2

3000  2013-08-23 22:59:59         0  ...          1               2.3

I am expecting a result that resembles the following:

     Timestamp       Max time gap                                            
2013-08-14 22:00:00    13.416600 
2013-08-14 22:10:00    14.088200    
2013-08-14 22:20:00    7.187153    
2013-08-14 22:30:00    16.444224      
2013-08-14 22:40:00    11.780500        
2013-08-14 22:50:00    12.051639        

Hope that I managed to be succinct and precise. I would really appreciate your help on this one!

Answer 1

If need maximum difference for each 10 minutes of data:

df['Timestamp'] = pd.to_datetime(df['Timestamp'])

df = (df.resample('10Min', on='Timestamp')['Timestamp']
        .apply(lambda x: x.diff().dt.total_seconds().max())
        .reset_index(name='Max time gap'))

print (df)
               Timestamp  Max time gap
0    2013-08-14 22:00:00          11.0
1    2013-08-14 22:10:00           NaN
2    2013-08-14 22:20:00           NaN
3    2013-08-14 22:30:00           NaN
4    2013-08-14 22:40:00           NaN
                 ...           ...
1297 2013-08-23 22:10:00           NaN
1298 2013-08-23 22:20:00           NaN
1299 2013-08-23 22:30:00           NaN
1300 2013-08-23 22:40:00           NaN
1301 2013-08-23 22:50:00           NaN

[1302 rows x 2 columns]

Test :

df['new'] = df.resample('10Min', on='Timestamp')['Timestamp'].diff()
print (df)
               Timestamp  Category  Class  Speed      new
0    2013-08-14 22:00:00         1      1    1.0      NaT
1    2013-08-14 22:00:01         1      2    1.0 00:00:01
2    2013-08-14 22:00:05         1      0    1.1 00:00:04
3    2013-08-14 22:00:07         1      1    1.2 00:00:02
4    2013-08-14 22:00:14         1      3    1.2 00:00:07
5    2013-08-14 22:00:15         1      0    1.2 00:00:01
6    2013-08-14 22:00:16         1      1    1.2 00:00:01
7    2013-08-14 22:00:27         1      2    1.2 00:00:11
8    2013-08-14 22:00:38         1      1    1.2 00:00:11
3000 2013-08-23 22:59:59         0      1    2.3      NaT

Answer 2

You can resample your data in every 10 minutes and apply an aggregate function to find the max time difference:

df.set_index(df.Timestamp.astype('datetime64'), inplace=True)
df['Timestamp'] = df['Timestamp'].astype('datetime64')

df['Timestamp'].resample('10m').agg(lambda x: np.max(x) - np.min(x))

Answer 3

Input dataset:

number,Timestamp,Category,Class,Speed
0 ,2013-08-14 22:00:00,1,1,1
1 ,2013-08-14 22:00:01,1,2,1
2 ,2013-08-14 22:00:05,1,0,1.1
3 ,2013-08-14 22:00:07,1,1,1.2
4 ,2013-08-14 22:00:14,1,3,1.2
5 ,2013-08-14 22:00:15,1,0,1.2
6 ,2013-08-14 22:00:16,1,1,1.2
7 ,2013-08-14 22:00:27,1,2,1.2
8 ,2013-08-14 22:00:38,1,1,1.2
8 ,2013-08-14 22:40:38,1,1,1.2
8 ,2013-08-14 22:45:38,1,1,1.2
8 ,2013-08-14 22:49:38,1,1,1.2
8 ,2013-08-14 22:50:38,1,1,1.2
8 ,2013-08-14 22:52:38,1,1,1.2
3000,2013-08-23 22:59:59,0,1,1
Process:

import pandas as pd
dataset = pd.read_csv('dataset.csv')  
dataset = pd.DataFrame(dataset)  
timestampField = pd.to_datetime(dataset['Timestamp'])  
startDate = pd.to_datetime('2013-08-14 22:00:00')  
episode = pd.Timedelta('10 minutes')  
maxInterval = pd.Timedelta('0 second')  
for index in range(1, len(timestampField)):  
  if timestampField[index] >= startDate + episode:  
      print(startDate, maxInterval.total_seconds())  
      startDate = startDate + episode  
      while timestampField[index] > startDate + episode:     
          startDate = startDate + episode
      maxInterval = pd.Timedelta('0 second')  
else:  
  localInterval = timestampField[index] -  timestampField[index - 1]  
  if localInterval > maxInterval:  
        maxInterval = localInterval

Output:
2013-08-14 22:00:00 11.0
2013-08-14 22:40:00 300.0
2013-08-14 22:50:00 120.0