有条件地合并 pandas DataFrame 中的行

Question

I have the following pandas DataFrame:

col1 col2                   col3        col4 
A    2021-03-28 01:40:00    1.381158    0.0
A    2021-03-28 01:50:00    0.480089    0.0
A    2021-03-28 03:00:00    0.000000    0.0
A    2021-03-28 03:00:00    0.111088    0.0
A    2021-03-28 03:10:00    0.000000    0.0
A    2021-03-28 03:10:00    0.000000    0.0
A    2021-03-28 03:10:00    0.151066    0.0
B    2021-03-28 03:10:00    1.231341    1.0

I need to merge rows that have the same col1 and col2 values, and take the non-zero value for col3 .

This is the expected output:

col1 col2                   col3        col4 
A    2021-03-28 01:40:00    1.381158    0.0
A    2021-03-28 01:50:00    0.480089    0.0
A    2021-03-28 03:00:00    0.111088    0.0
A    2021-03-28 03:10:00    0.151066    0.0
B    2021-03-28 03:10:00    1.231341    1.0

How can I do this merging?

Answer 1

We can use groupby + idxmax for this:

idx = df.groupby(["col1", "col2"])["col3"].idxmax().to_numpy()
df.loc[idx]

  col1                 col2  col3  col4
0    A  2021-03-28 01:40:00  1.38  0.00
1    A  2021-03-28 01:50:00  0.48  0.00
3    A  2021-03-28 03:00:00  0.11  0.00
6    A  2021-03-28 03:10:00  0.15  0.00
7    B  2021-03-28 03:10:00  1.23  1.00

Answer 2

An option via sort_values + drop_duplicates with keep='last':

(If the DataFrame is already sorted as it is in the question then just drop_duplicates and keep last)

df = df.sort_values(['col1', 'col2', 'col3']) \
    .drop_duplicates(['col1', 'col2'], keep='last')

df :

 col1                col2      col3  col4
0    A 2021-03-28 01:40:00  1.381158   0.0
1    A 2021-03-28 01:50:00  0.480089   0.0
3    A 2021-03-28 03:00:00  0.111088   0.0
6    A 2021-03-28 03:10:00  0.151066   0.0
7    B 2021-03-28 03:10:00  1.231341   1.0

Answer 3

We can groupby col1 and col2 and get the non-zero values from col3 .

Code

df.col2 = pd.to_datetime(df.col2) ## if required
df[df.groupby(['col1', 'col2'])['col3'].transform(lambda x: x != 0)]

Output

    col1    col2            col3        col4
0   A   2021-03-28 01:40:00 1.381158    0.0
1   A   2021-03-28 01:50:00 0.480089    0.0
2   A   2021-03-28 03:00:00 0.111088    0.0
3   A   2021-03-28 03:10:00 0.151066    0.0
4   B   2021-03-28 03:10:00 1.231341    1.0