在熊猫中查找重复项的最快方法

Question

I have a dataframe like this:

date             IP                 date_2            IP_2
2020-02-17       81.195.104.48      2020-02-24        219.85.238.142
2020-02-17       83.71.247.175      2020-02-24        187.134.23.124
2020-02-17       83.71.247.175      Nat               NaN

I am trying to get duplicates when comparing IP and IP_2 values. IP has more rows than IP_2 hence I am checking if IP_2 exsist in IP like so:

df['duplicates']=df['IP_2'].isin(df['IP'])

Is there a faster way of getting only the duplicated df rather than adding new column that checks .isin() method? Desired output would be a new dataframe holding only the duplicated values.

Thank you for your suggestions.

Answer 1

Set comparison seems to me the fastest way:

set_common = set(df['IP']) & set(df['IP_2'])

PS Another way is to actually play with IP format (ie turn it into integer, then do some kind of comparison, but this apparently would make sense for a very big table).