Obtain a column average of only those rows with a fuzzy ratio above a given value

Question

I am trying to obtain, for each value in a grouped pandas column, the average value of another column with 1 and 0 only of those rows where the fuzz.partial_ratio() of the column is above a given limit (say above 80).

I know this might be unclear, so here is an example of my data

col1 col2      col3
A    Miami       1
A    Miami       0
A    Miami.      0
A    Barcelona   0
A    Barc elona  0
A    Shanghai    1
A    Shangai     0
B    Miami       1
B    Miami       1
B    Miami.      1
B    Barcelona   0
B    Barc elona  0
B    Shanghai    1
B    Shangai    0

I am trying to groupby('col1') and for each value in col2 estimate in a new column the average of col3 of only those rows where the fuzzy_ratio of col2 is above 80.

For example, in row 0 , df['col2']='Miami' . Then, I want to obtain the fuzzy_ratio() of 'Miami' with all the values in col2 with df['col1']='A' , and obtain the average of col3 of those rows where the ratio is >80 and store it in a new column. This will be rows 1 and 2 , which is 0. Same for row 2 , but in this case, the average will be 0.5.

The output I am trying to get would look like this:

col1 col2      col3 col4
A    Miami       1   0.33
A    Miami       0   0.33
A    Miami.      0   0.33
A    Barcelona   0   0
A    Barc elona  0   0
A    Shanghai    1   0.5
A    Shangai     0   0.5
B    Miami       1   1
B    Miami       1   1
B    Miami.      1   1
B    Barcelona   0   0
B    Barc elona  0   0
B    Shanghai    1   0.5
B    Shangai     0   0.5

I managed to do it with a for loop for each value in col2 , but I have a relatively large dataset (+10 million rows) and it would take forever.

Is there any way I can avoid that for loop to do this task?

Thanks a lot!!!!!

Answer 1

This is not efficient but i think will do what you need

from fuzzywuzzy import fuzz
import pandas as pd
import numpy as np

# helper function
def remove_element(lst, index):
    "Removes an element from a list based on the index"
    tmp = lst.copy()
    del tmp[index]
    return tmp


df = pd.DataFrame({'col1':['A', 'A', 'A', 'A', 'A', 'A', 'A', 
                           'B', 'B', 'B', 'B', 'B', 'B', 'B'], 
                  'col2':['Miami', 'Miami', 'Miami.', 'Barcelona', 'Barc elona', 
                         'Shanghai', 'Shangai', 'Miami', 'Miami', 'Miami.', 
                         'Barcelona', 'Barc elona', 'Shanghai', 'Shangai'], 
                  'col3':[1, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0]})

# create a column that indicates the index of the element within the group
df['col2_index'] = 1
df['col2_index'] = df.groupby('col1')['col2_index'].cumsum() - 1

# create a list of the elements within the group
df['col2_list'] = df['col1'].map(df.groupby('col1')['col2'].apply(list))
df['col3_list'] = df['col1'].map(df.groupby('col1')['col3'].apply(list))

# remove the element associated with col2 and col3 respectively
df['col2_list'] = df.apply(lambda x: remove_element(x['col2_list'], x['col2_index']), axis=1)
df['col3_list'] = df.apply(lambda x: remove_element(x['col3_list'], x['col2_index']), axis=1)

# apply the threshold of 80 for the partial_ratio
df['key'] = df.apply(lambda x: 
         np.array([fuzz.partial_ratio(x['col2'], el) for el in x['col2_list']]) >= 80, axis=1)

# get the average of col3 for those that pass the threshold
df['result'] = df.apply(lambda x: np.mean(np.array(x['col3_list'])[x['key']]), axis=1)

df

 col1   col2       col3 col2_index  col2_list                                           col3_list           key                                      result
0   A   Miami       1   0           [Miami, Miami., Barcelona, Barc elona, Shangha...   [0, 0, 0, 0, 1, 0]  [True, True, False, False, False, False]    0.0
1   A   Miami       0   1           [Miami, Miami., Barcelona, Barc elona, Shangha...   [1, 0, 0, 0, 1, 0]  [True, True, False, False, False, False]    0.5
2   A   Miami.      0   2           [Miami, Miami, Barcelona, Barc elona, Shanghai...   [1, 0, 0, 0, 1, 0]  [True, True, False, False, False, False]    0.5
3   A   Barcelona   0   3           [Miami, Miami, Miami., Barc elona, Shanghai, S...   [1, 0, 0, 0, 1, 0]  [False, False, False, True, False, False]   0.0
4   A   Barc elona  0   4           [Miami, Miami, Miami., Barcelona, Shanghai, Sh...   [1, 0, 0, 0, 1, 0]  [False, False, False, True, False, False]   0.0
5   A   Shanghai    1   5           [Miami, Miami, Miami., Barcelona, Barc elona, ...   [1, 0, 0, 0, 0, 0]  [False, False, False, False, False, True]   0.0
6   A   Shangai     0   6           [Miami, Miami, Miami., Barcelona, Barc elona, ...   [1, 0, 0, 0, 0, 1]  [False, False, False, False, False, True]   1.0
7   B   Miami       1   0           [Miami, Miami., Barcelona, Barc elona, Shangha...   [1, 1, 0, 0, 1, 0]  [True, True, False, False, False, False]    1.0
8   B   Miami       1   1           [Miami, Miami., Barcelona, Barc elona, Shangha...   [1, 1, 0, 0, 1, 0]  [True, True, False, False, False, False]    1.0
9   B   Miami.      1   2           [Miami, Miami, Barcelona, Barc elona, Shanghai...   [1, 1, 0, 0, 1, 0]  [True, True, False, False, False, False]    1.0
10  B   Barcelona   0   3           [Miami, Miami, Miami., Barc elona, Shanghai, S...   [1, 1, 1, 0, 1, 0]  [False, False, False, True, False, False]   0.0
11  B   Barc elona  0   4           [Miami, Miami, Miami., Barcelona, Shanghai, Sh...   [1, 1, 1, 0, 1, 0]  [False, False, False, True, False, False]   0.0
12  B   Shanghai    1   5           [Miami, Miami, Miami., Barcelona, Barc elona, ...   [1, 1, 1, 0, 0, 0]  [False, False, False, False, False, True]   0.0
13  B   Shangai     0   6           [Miami, Miami, Miami., Barcelona, Barc elona, ...   [1, 1, 1, 0, 0, 1]  [False, False, False, False, False, True]   1.0

For the updated question just remove the part of the code that refines the lists

df = pd.DataFrame({'col1':['A', 'A', 'A', 'A', 'A', 'A', 'A', 
                           'B', 'B', 'B', 'B', 'B', 'B', 'B'], 
                  'col2':['Miami', 'Miami', 'Miami.', 'Barcelona', 'Barc elona', 
                         'Shanghai', 'Shangai', 'Miami', 'Miami', 'Miami.', 
                         'Barcelona', 'Barc elona', 'Shanghai', 'Shangai'], 
                  'col3':[1, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0]})


df['col2_list'] = df['col1'].map(df.groupby('col1')['col2'].apply(list))
df['col3_list'] = df['col1'].map(df.groupby('col1')['col3'].apply(list))

df['key'] = df.apply(lambda x: 
         np.array([fuzz.partial_ratio(x['col2'], el) for el in x['col2_list']]) >= 80, axis=1)

df['result'] = df.apply(lambda x: np.mean(np.array(x['col3_list'])[x['key']]), axis=1)

df

  col1  col2     col3   col2_list                                           col3_list               key                                             result
0   A   Miami       1   [Miami, Miami, Miami., Barcelona, Barc elona, ...   [1, 0, 0, 0, 0, 1, 0]   [True, True, True, False, False, False, False]  0.333333
1   A   Miami       0   [Miami, Miami, Miami., Barcelona, Barc elona, ...   [1, 0, 0, 0, 0, 1, 0]   [True, True, True, False, False, False, False]  0.333333
2   A   Miami.      0   [Miami, Miami, Miami., Barcelona, Barc elona, ...   [1, 0, 0, 0, 0, 1, 0]   [True, True, True, False, False, False, False]  0.333333
3   A   Barcelona   0   [Miami, Miami, Miami., Barcelona, Barc elona, ...   [1, 0, 0, 0, 0, 1, 0]   [False, False, False, True, True, False, False] 0.000000
4   A   Barc elona  0   [Miami, Miami, Miami., Barcelona, Barc elona, ...   [1, 0, 0, 0, 0, 1, 0]   [False, False, False, True, True, False, False] 0.000000
5   A   Shanghai    1   [Miami, Miami, Miami., Barcelona, Barc elona, ...   [1, 0, 0, 0, 0, 1, 0]   [False, False, False, False, False, True, True] 0.500000
6   A   Shangai     0   [Miami, Miami, Miami., Barcelona, Barc elona, ...   [1, 0, 0, 0, 0, 1, 0]   [False, False, False, False, False, True, True] 0.500000
7   B   Miami       1   [Miami, Miami, Miami., Barcelona, Barc elona, ...   [1, 1, 1, 0, 0, 1, 0]   [True, True, True, False, False, False, False]  1.000000
8   B   Miami       1   [Miami, Miami, Miami., Barcelona, Barc elona, ...   [1, 1, 1, 0, 0, 1, 0]   [True, True, True, False, False, False, False]  1.000000
9   B   Miami.      1   [Miami, Miami, Miami., Barcelona, Barc elona, ...   [1, 1, 1, 0, 0, 1, 0]   [True, True, True, False, False, False, False]  1.000000
10  B   Barcelona   0   [Miami, Miami, Miami., Barcelona, Barc elona, ...   [1, 1, 1, 0, 0, 1, 0]   [False, False, False, True, True, False, False] 0.000000
11  B   Barc elona  0   [Miami, Miami, Miami., Barcelona, Barc elona, ...   [1, 1, 1, 0, 0, 1, 0]   [False, False, False, True, True, False, False] 0.000000
12  B   Shanghai    1   [Miami, Miami, Miami., Barcelona, Barc elona, ...   [1, 1, 1, 0, 0, 1, 0]   [False, False, False, False, False, True, True] 0.500000
13  B   Shangai     0   [Miami, Miami, Miami., Barcelona, Barc elona, ...   [1, 1, 1, 0, 0, 1, 0]   [False, False, False, False, False, True, True] 0.500000