熊猫使用条件删除重复项

Question

Say I have the following dataframe:

>>> import pandas as pd
>>> 
>>> d=pd.DataFrame()
>>> 
>>> d['Var1']=['A','A','B','B','C','C','D','E','F']
>>> d['Var2']=['A','Z','B','Y','X','C','Q','N','P']
>>> d['Value']=[34, 45, 23, 54, 65, 77,100,102,44]
>>> d
  Var1 Var2  Value
0    A    A     34
1    A    Z     45
2    B    B     23
3    B    Y     54
4    C    X     65
5    C    C     77
6    D    Q    100
7    E    N    102
8    F    P     44
>>> 

I want to drop cases where there are duplicates in "Var1", but I want to make sure that the duplicate that is kept is the one where 'Var1'=='Var2'

My output dataframe would be:

     Var2  Value
Var1            
A       A     34
B       B     23
C       C     77
D       Q    100
E       N    102
F       P     44
>>> 

Any suggestions as to how I can do this? Would using groupby filter be the best approach?

Answer 1

Here's a one-liner:

>>> d.loc[~d.Var1[(d.Var1 == d.Var2).argsort()].duplicated('last')]

  Var1 Var2  Value
0    A    A     34
2    B    B     23
5    C    C     77
6    D    Q    100
7    E    N    102
8    F    P     44

You can then set the index on Var1 if you want ( d.set_index('Var1') ) to get exactly the output you posted.

To break it down:

• d.Var1[(d.Var1 == d.Var2).argsort()] is series with values in Var1 arranged in such a way that the rows where Var1 == Var2 are at the end

• ~d.Var1[(d.Var1 == d.Var2).argsort()].duplicated('last') is true for rows where Var1 is non-duplicated; if there are duplicates we pick the last one (so Var1 == Var2 has priority)

Answer 2

My suggestion would be to create Var 2 and the Value as a dictionary.

    d['Var1']=['A','A','B','B','C','C','D','E','F']
    d['Var2']=['A','Z','B','Y','X','C','Q','N','P']
    d['Var2Val'] = {'A':34,'Z':45,'B':23,'Y':54,'X':65,'C':77,'Q':100,'N':102,'P':44}

Then I would create a list for Var1 without duplicates, print those if they are in Var2

    for x in d['Var1']:
        if d['Var1'][x] in d['Var2']:

then print the table

at least this would be the simplest way, even though it might be a little long