Python Pandas：在循环中比较两个列表的最有效方法是什么？

Question

I have a ground truth dataset 'gt' (with 100 entries) which looks like this:

    org_o                  shh group
    ArabsGate               1   1
    ArabsGate Company       1   1
    AS EMT                 NaN  2
    AS EMT Mobile Internet  1   2
    DigitalEffex (MH)      NaN  3
    DigitalEffex            1   3
    Aruba S.p.A.            1   4
    Aruba S.p.              1   4

and I would like to compare it to a huge dataframe 'df' which looks like this:

        match           org_o 
        as emt        AS EMT                   
        as emt        AS EMT Mobile Internet    
        digitaleffex  DigitalEffex (MH)    
        digitaleffex  DigitalEffex
        digitaleffex  Digital

As a result of comparision I want to if the same group with the same org_o exists in my df or not. So for each group both counts or members of the group, and actual org_o names. So for instance where we have both 'Aruba SpA' and 'Aruba Sp' in df and wether they are matched to a same keyword ('match' column) in one group.

Here is what I did, but is not really what I am looking for.

gt.groupby('group').count()['org_o']
df.merge(gt, on  = 'org_o')

Eventually I would like to count false positive/negatives. this is the expected output:

        match           org_o                 tag
        as emt        AS EMT                   TP
        as emt        AS EMT Mobile Internet   TP   
        digitaleffex  DigitalEffex (MH)        TP
        digitaleffex  DigitalEffex             TP
        digitaleffex  Digital                  FP

Can anybody help with it?

Answer 1

Looks like what you need is simple lookup -

df['tag'] = np.where(df['org_o'].isin(gt['org_o']), 'TP', 'FP')

Here we are adding a new column tag to the df . We are using numpy's where function to check if the org_o in df is present in gt . If yes, then assign TP as the value of the tag to that row, otherwise assign FP .

As far as efficiency is concerned, this "lookup" is fairly efficient, because when using isin , pandas will convert the values to compare (in this case gt['org_o'] ) into a set , so the lookup time will be O(n * log m)

Answer 2

Here's one way to do it.

Assign the tag column initially with 'FP'

In [4]: df['tag'] = 'FP'

Filter out rows with gt['org_o'] values in df['org_o'] using df['org_o'].isin(gt['org_o'])

And, assign the tag column with TP

In [5]: df.loc[df['org_o'].isin(gt['org_o']), 'tag'] = 'TP'

In [6]: df
Out[6]:
          match                   org_o tag
0        as emt                  AS EMT  TP
1        as emt  AS EMT Mobile Internet  TP
2  digitaleffex       DigitalEffex (MH)  TP
3  digitaleffex            DigitalEffex  TP
4  digitaleffex                 Digital  FP

I find @Shashank's answer elegant. A minor addition would be in case, if gt['org_o'] has repetitive values, you can take unique array instead.

df['tag'] = np.where(df['org_o'].isin(gt['org_o'].unique()), 'TP', 'FP')