计算两个熊猫数据框的字符串之间的距离

Question

I have 2 data frames:

df1:
Date       Name   Num  
2013-11-24 Banana 22.1 
2013-11-24 Orange  8.6 
2013-11-24 Apple   7.6 
2013-11-24 Celery 10.2 

df2:
Date       Name   Num  
2013-11-24 Celery 22.1 
2013-11-24 0r@nge  8.6 
2013-11-24 @ppl3   7.6 
2013-11-24 BananaX 10.2 

I want to find similar rows, for that, I need to find similarity of Name between 2 data frame right now I am iterating each data frame and compute similarity with all the other rows of the other data frame (which is very time consuming) and find the maximum value and if it was greater than certain threshold I will do something with it.

dfResult = pd.DataFrame()
import pandas as pd
from fuzzywuzzy import fuzz
for indexD, rowD in dfD.iterrows():
    for indexS, rowS in dfS.iterrows():
        data = pd.DataFrame({"ratio": fuzz.token_set_ratio(rowD['Name'], rowS['Name']),
                             "indexD": rowD['Num'], "indexS": rowS['Num']}, index=[indexS])
    maxMatch = dfTMP.loc[dfTMP['ratio'].idxmax()]
    ......
    ......
    ......
    resultMatch = create_match_row(maxMatch, dfD, dfS)

After each iteration I am getting

indexD      1
indexS      4
ratio     100
Name: 3, dtype: int64
1
indexD     2
indexS     1
ratio     35
Name: 0, dtype: int64
2
indexD     3
indexS     3
ratio     45
Name: 2, dtype: int64
3
indexD     4
indexS     4
ratio     33
Name: 3, dtype: int64

which the max function should return :

    indexD      1
    indexS      4
    ratio     100

Which means row 1 from data frame 1 is similar to row 4 in data frame 2.

I wanted to know is there any better way so I can compute the distance in one shot and remove the inner loop? and find the best match for each row ( name ) in the first data frame with the second data frame?

Expected output: for each row in data frame one I like to get the data frame (just a simple index) that shows which row in data frame 2 is the most identical one.

Answer 1

IIUIC, Here's one way

In [3456]: def get_fuzz(df, w):
      ...:     s = df['Name'].apply(lambda y: fuzz.token_set_ratio(y, w))
      ...:     idx = s.idxmax()
      ...:     return {'name': df['Name'].iloc[idx], 'index': idx, 'val': s.max()}
      ...:

In [3457]: df1['Name'].apply(lambda x: get_fuzz(df2, x))
Out[3457]:
0    {u'index': 3, u'name': u'BananaX', u'val': 92}
1     {u'index': 1, u'name': u'0r@nge', u'val': 67}
2      {u'index': 2, u'name': u'@ppl3', u'val': 67}
3    {u'index': 0, u'name': u'Celery', u'val': 100}
Name: Name, dtype: object

assign the result to df1 , if you need

In [3458]: df1.assign(search=df1['Name'].apply(lambda x: get_fuzz(df2, x)))
Out[3458]:
         Date    Name   Num                                          search
0  2013-11-24  Banana  22.1  {u'index': 3, u'name': u'BananaX', u'val': 92}
1  2013-11-24  Orange   8.6   {u'index': 1, u'name': u'0r@nge', u'val': 67}
2  2013-11-24   Apple   7.6    {u'index': 2, u'name': u'@ppl3', u'val': 67}
3  2013-11-24  Celery  10.2  {u'index': 0, u'name': u'Celery', u'val': 100}

---

Details

In [3459]: df1
Out[3459]:
         Date    Name   Num
0  2013-11-24  Banana  22.1
1  2013-11-24  Orange   8.6
2  2013-11-24   Apple   7.6
3  2013-11-24  Celery  10.2

In [3460]: df2
Out[3460]:
         Date     Name   Num
0  2013-11-24   Celery  22.1
1  2013-11-24   0r@nge   8.6
2  2013-11-24    @ppl3   7.6
3  2013-11-24  BananaX  10.2