我如何知道熊猫数据框中的重复行？

Question

I am working with Pandas and the function duplicated() to detect which rows are equal:

import pandas as pd

d = {
    1: {'name': 'n1', 1: 10, 2: 20, 3: 30},
    2: {'name': 'n2', 1: 10, 2: 20, 3: 30},
    3: {'name': 'n3', 1: 11, 2: 21, 3: 30},
    4: {'name': 'n4', 1: 11, 2: 21, 3: 30},
    5: {'name': 'n5', 1: 12, 2: 22, 3: 30},
    6: {'name': 'n6', 1: 13, 2: 22, 3: 30},
    7: {'name': 'n7', 1: 14,        3: 35},
    8: {'name': 'n8',        2: 22, 3: 35},
}
pd.DataFrame.from_dict(d).transpose().set_index('name')

This gives me a nice data frame like this one:

          1    2   3
name              
n1       10   20  30    # same as n2
n2       10   20  30    # same as n1
n3       11   21  30    # same as n4
n4       11   21  30    # same as n3
n5       12   22  30
n6       13   22  30
n7       14  NaN  35
n8      NaN   22  35

Now I want to group those lines whose columns are the same. That is, I want Pandas to tell me that the rows n1 and n2 are equal, and so n3 and n4 are.

Using duplicated() I get some interesting results:

df[df.duplicated(keep=False)]
         1   2   3
name            
n1      10  20  30
n2      10  20  30
n3      11  21  30
n4      11  21  30

Which is correct, since it shows those rows that have duplicated. However, my aim is to get to know which are those columns, as well as knowing which are the tuples of duplicates. That is, I would need a result on the form of [(n1, n2), (n3,n4)] , a list with the duplicates one to each other. List, dict, anything works to me as well as it has this info.

I have been browsing through Pandas' documentation and cannot find something like this. I tried a bit with groupby() , but nothing reasonable comes up.

Answer 1

You can use groupby by all columns and convert indices to list for each group, last convert Series to list:

df1 = df[df.duplicated(keep=False)]

df1 = df1.groupby(df1.columns.tolist()).apply(lambda x: x.index.tolist()).values.tolist()
print (df1)
[['n1', 'n2'], ['n3', 'n4']]

Detail:

print (df1.groupby(df1.columns.tolist()).apply(lambda x: x.index.tolist()))
1   2   3 
10  20  30    [n1, n2]
11  21  30    [n3, n4]
dtype: object