使用 pandas 在时间序列 dataframe 中删除 dataframe 行

Question

I have the below sequence of data as a pandas dataframe

id,start,end,duration
303,2012-06-25 17:59:43,2012-06-25 18:01:29,105
404,2012-06-25 18:01:29,2012-06-25 18:01:55,25
303,2012-06-25 18:01:56,2012-06-25 18:02:06,10
303,2012-06-25 18:02:23,2012-06-25 18:02:44,21
404,2012-06-25 18:02:45,2012-06-25 18:02:51,6
303,2012-06-25 18:02:54,2012-06-25 18:03:17,23
404,2012-06-25 18:03:24,2012-06-25 18:03:41,17
303,2012-06-25 18:03:43,2012-06-25 18:05:51,128
101,2012-06-25 18:05:58,2012-06-25 18:24:22,1104
404,2012-06-25 18:24:24,2012-06-25 18:25:25,61
101,2012-06-25 18:25:25,2012-06-25 18:25:462,21
404,2012-06-25 18:25:49,2012-06-25 18:26:00,11
101,2012-06-25 18:26:01,2012-06-25 18:26:04,3
404,2012-06-25 18:26:05,2012-06-25 18:28:49,164
202,2012-06-25 18:28:52,2012-06-25 18:28:57,5
404,2012-06-25 18:29:00,2012-06-25 18:29:24,24

It should always be the case that id 404 gets repeated after another different id.

For example if the above is motion sensors in a house eg 404 :hallway, 202 :bedroom, 303 :kitchen, 201 :studyroom, where the hallway is in the middle, then moving from bedroom to kitchen to studyroom and back to bedroom should trigger 202 , 404 , 303 , 404 , 201 , 404 , 202 in that order because one always passes through the hallway (404) to any room. My output has cases that violate this sequence and I want to drop such rows.

For example from the snippet dataframe above the below rows violate this:

303,2012-06-25 18:01:56,2012-06-25 18:02:06,10
303,2012-06-25 18:02:23,2012-06-25 18:02:44,21

303,2012-06-25 18:03:43,2012-06-25 18:05:51,128
101,2012-06-25 18:05:58,2012-06-25 18:24:22,1104

and therefore the rows below should be droped (but of course I have a much larger dataset).

303,2012-06-25 18:02:23,2012-06-25 18:02:44,21
101,2012-06-25 18:05:58,2012-06-25 18:24:22,1104

I have tried shift and drop but the result still has some inconsistencies.

df['id_ns'] = df['id'].shift(-1)
df['id_ps'] = df['id'].shift(1)

if (df['id'] != 404):
    df.drop(df[(df.id_ns != 404) & (df.id_ps != 404)].index, axis=0, inplace=True)

How best can I approach this?

Answer 1

Use Series.ne + Series.shift along with optional parameter fill_value to create a boolean mask , use this mask to filter/drop the rows:

mask = df['id'].ne(404) & df['id'].shift(fill_value=404).ne(404)
df = df[~mask]

Result:

print(df)
     id                start                  end  duration
0   303  2012-06-25 17:59:43  2012-06-25 18:01:29       105
1   404  2012-06-25 18:01:29  2012-06-25 18:01:55        25
2   303  2012-06-25 18:01:56  2012-06-25 18:02:06        10
4   404  2012-06-25 18:02:45  2012-06-25 18:02:51         6
5   303  2012-06-25 18:02:54  2012-06-25 18:03:17        23
6   404  2012-06-25 18:03:24  2012-06-25 18:03:41        17
7   303  2012-06-25 18:03:43  2012-06-25 18:05:51       128
9   404  2012-06-25 18:24:24  2012-06-25 18:25:25        61
10  101  2012-06-25 18:25:25  2012-06-25 18:25:46        21
11  404  2012-06-25 18:25:49  2012-06-25 18:26:00        11
12  101  2012-06-25 18:26:01  2012-06-25 18:26:04         3
13  404  2012-06-25 18:26:05  2012-06-25 18:28:49       164
14  202  2012-06-25 18:28:52  2012-06-25 18:28:57         5
15  404  2012-06-25 18:29:00  2012-06-25 18:29:24        24