熊猫有选择地删除连续的重复项

Question

我一直在查看有关如何在 Pandas 数据框中有选择地删除连续重复项的所有问题/答案，但仍然无法弄清楚以下情况：

import pandas as pd
import numpy as np

def random_dates(start, end, n, freq, seed=None):
    if seed is not None:
        np.random.seed(seed)

    dr = pd.date_range(start, end, freq=freq)
    return pd.to_datetime(np.sort(np.random.choice(dr, n, replace=False)))

date = random_dates('2018-01-01', '2018-01-12', 20, 'H', seed=[3, 1415])

data = {'Timestamp': date, 
        'Message': ['Message received.','Sending...', 'Sending...', 'Sending...', 'Work in progress...', 'Work in progress...', 
                    'Message received.','Sending...', 'Sending...','Work in progress...',
                    'Message received.','Sending...', 'Sending...', 'Sending...','Work in progress...', 'Work in progress...', 'Work in progress...',
                    'Message received.','Sending...', 'Sending...']}

df = pd.DataFrame(data, columns = ['Timestamp', 'Message'])

我有以下数据框：

             Timestamp              Message
0  2018-01-02 03:00:00    Message received.
1  2018-01-02 11:00:00           Sending...
2  2018-01-03 04:00:00           Sending...
3  2018-01-04 11:00:00           Sending...
4  2018-01-04 16:00:00  Work in progress...
5  2018-01-04 17:00:00  Work in progress...
6  2018-01-05 05:00:00    Message received.
7  2018-01-05 11:00:00           Sending...
8  2018-01-05 17:00:00           Sending...
9  2018-01-06 02:00:00  Work in progress...
10 2018-01-06 14:00:00    Message received.
11 2018-01-07 07:00:00           Sending...
12 2018-01-07 20:00:00           Sending...
13 2018-01-08 01:00:00           Sending...
14 2018-01-08 02:00:00  Work in progress...
15 2018-01-08 15:00:00  Work in progress...
16 2018-01-09 00:00:00  Work in progress...
17 2018-01-10 03:00:00    Message received.
18 2018-01-10 09:00:00           Sending...
19 2018-01-10 14:00:00           Sending...

我想删除 df['Message'] 列中的连续重复项，仅当 'Message' 为 'Work in progress...' 并保留第一个实例（此处例如需要删除索引 5、15 和 16），理想情况下我想得到：

             Timestamp              Message
0  2018-01-02 03:00:00    Message received.
1  2018-01-02 11:00:00           Sending...
2  2018-01-03 04:00:00           Sending...
3  2018-01-04 11:00:00           Sending...
4  2018-01-04 16:00:00  Work in progress...
6  2018-01-05 05:00:00    Message received.
7  2018-01-05 11:00:00           Sending...
8  2018-01-05 17:00:00           Sending...
9  2018-01-06 02:00:00  Work in progress...
10 2018-01-06 14:00:00    Message received.
11 2018-01-07 07:00:00           Sending...
12 2018-01-07 20:00:00           Sending...
13 2018-01-08 01:00:00           Sending...
14 2018-01-08 02:00:00  Work in progress...
17 2018-01-10 03:00:00    Message received.
18 2018-01-10 09:00:00           Sending...
19 2018-01-10 14:00:00           Sending...

我尝试过类似帖子中提供的解决方案，例如：

df['Message'].loc[df['Message'].shift(-1) != df['Message']]

我还计算了消息的长度：

df['length'] = df['Message'].apply(lambda x: len(x))

并写了一个有条件的丢弃：

df.loc[(df['length'] ==17) | (df['length'] ==10) | ~df['Message'].duplicated(keep='first')]

它看起来更好，但仍然完全删除了索引 14、15 和 16，因此行为不端，请参阅：

             Timestamp              Message  length
0  2018-01-02 03:00:00    Message received.      17
1  2018-01-02 11:00:00           Sending...      10
2  2018-01-03 04:00:00           Sending...      10
3  2018-01-04 11:00:00           Sending...      10
4  2018-01-04 16:00:00  Work in progress...      19
6  2018-01-05 05:00:00    Message received.      17
7  2018-01-05 11:00:00           Sending...      10
8  2018-01-05 17:00:00           Sending...      10
10 2018-01-06 14:00:00    Message received.      17
11 2018-01-07 07:00:00           Sending...      10
12 2018-01-07 20:00:00           Sending...      10
13 2018-01-08 01:00:00           Sending...      10
17 2018-01-10 03:00:00    Message received.      17
18 2018-01-10 09:00:00           Sending...      10
19 2018-01-10 14:00:00           Sending...      10

感谢您的时间和帮助！

Answer 1

首先使用按Series.shift比较和链掩码过滤第一个连续值，并过滤所有没有进行Work in progress...行Work in progress...值：

df = df[(df['Message'].shift() != df['Message']) | (df['Message'] != 'Work in progress...')]
print (df)
             Timestamp              Message
0  2018-01-02 03:00:00    Message received.
1  2018-01-02 11:00:00           Sending...
2  2018-01-03 04:00:00           Sending...
3  2018-01-04 11:00:00           Sending...
4  2018-01-04 16:00:00  Work in progress...
6  2018-01-05 05:00:00    Message received.
7  2018-01-05 11:00:00           Sending...
8  2018-01-05 17:00:00           Sending...
9  2018-01-06 02:00:00  Work in progress...
10 2018-01-06 14:00:00    Message received.
11 2018-01-07 07:00:00           Sending...
12 2018-01-07 20:00:00           Sending...
13 2018-01-08 01:00:00           Sending...
14 2018-01-08 02:00:00  Work in progress...
17 2018-01-10 03:00:00    Message received.
18 2018-01-10 09:00:00           Sending...
19 2018-01-10 14:00:00           Sending...

Answer 2

您可以首先获取所有带有“正在进行的工作”的消息，并将它们与前一个元素进行比较，然后进行过滤：

condition = (df['Message'] == 'Work in progress...') & (df['Message']==df['Message'].shift(1))

df[~condition]

     Timestamp           Message
0   2018-01-02 03:00:00 Message received.
1   2018-01-02 11:00:00 Sending...
2   2018-01-03 04:00:00 Sending...
3   2018-01-04 11:00:00 Sending...
4   2018-01-04 16:00:00 Work in progress...
6   2018-01-05 05:00:00 Message received.
7   2018-01-05 11:00:00 Sending...
8   2018-01-05 17:00:00 Sending...
9   2018-01-06 02:00:00 Work in progress...
10  2018-01-06 14:00:00 Message received.
11  2018-01-07 07:00:00 Sending...
12  2018-01-07 20:00:00 Sending...
13  2018-01-08 01:00:00 Sending...
14  2018-01-08 02:00:00 Work in progress...
17  2018-01-10 03:00:00 Message received.
18  2018-01-10 09:00:00 Sending...
19  2018-01-10 14:00:00 Sending...