Set Dataframe values to values in another Dataframe

Question

I am attempting to merge two dataframes about movies on their title columns. Unfortunately they don't share indices, are ordered differently, and do not contain all the same movies. I have cleaned up the titles enough to the point where I can compare them and determine whether they are acceptably similar to be called the same movie. Following that comparison I am trying to set those "matching" titles equal to eachother (taking the title from df2 and overwriting it's equivalent in df), but I can't seem to get it to work.

This is df.head(10)

      Unnamed: 0                                       title    imdb_id  \
72            72                                   Gully Boy  tt2395469   
767          767                                   Long Shot  tt2139881   
1000        1000                                      Little  tt8085790   
1285        1285                                       Dumbo  tt3861390   
1342        1342                           Don"t Stop Me Now  tt9260446   
1358        1358  How to Train Your Dragon: The Hidden World  tt2386490   
1621        1621                               Captive State  tt5968394   
1658        1658                   Spider-Man: Far from Home  tt6320628   
1944        1944                           Avengers: Endgame  tt4154796   
1946        1946                              Captain Marvel  tt4154664   

This is df2.head(10)

      Unnamed: 0  bo_year_rank                                      title  \
7455           0             1                           Avengers Endgame   
7456           1             2                              The Lion King   
7457           2             3                            Captain Marvel    
7458           3             4                   Spider-Man Far from Home   
7459           4             5                                Toy Story 4   
7460           5             6                                    Aladdin   
7461           6             7         Fast  Furious Presents Hobbs  Shaw   
7462           7             8                        The Wandering Earth   
7463           8             9                                     Ne Zha   
7464           9            10  How to Train Your Dragon The Hidden World   

The comparison function I am using:

def fix_stupid_titles(title, list_titles, min_score=0):
    # -1 score incase we don't get any matches
    max_score = -1
    # Returning empty name for no match as well
    max_title = ""
    # Iterating over all names in the other
    for title2 in list_titles:
        #Finding fuzzy match score
        score = fuzz.ratio(title, title2)
        # Checking if we are above our threshold and have a better score
        if (score > min_score) & (score > max_score):
            max_title = title2
            max_score = score

    return (title, max_title, max_score)

My thought is to iterate through one dataframe and apply the stupid_titles function to check similarity, but then I'm not sure how to update the titles so they are equal.

for i, title in enumerate(df.title):
    thing=fix_stupid_titles(title, df2.title, min_score=75)
    #thing saves the return of the functoin (title, max_title, score)
    df['title'][thing[1]]=thing[0]

I want the titles in df to be overwritten with the titles from df2 when the function determines they are similar.

Any help would be appreciated. Or suggestions on how to go about this more efficiently. Thanks!

Answer 1

You can simply achieve it with the process of fuzzywuzzy.

df_choices = df['title'].values

This will get the list of title from your df Dataframe

def fuzzratio(row, options):
    match, score = process.extractOne(row['title'], options, scorer=fuzz.ratio)
    row['matches'] = match
    row['score'] = score
    return row

matches = df2.apply(lambda x: fuzzratio(x, df_choices), axis=1)
matches.columns = ['title', 'matches', 'score']

The function fuzzratio is used to get best match title for all titles in your df2 Dataframe.

merges = df2.merge(matches, on='title', how = 'inner')
merges = merges.loc[merges['score'] >= 90]
merges.drop('score', axis=1, inplace=True)
merges.columns = ['new_title', 'title']

Filter all the title which have score greater than 75.

df = df.merge(merges, on='title', how = 'left')
df.loc[~df['new_title'].isna(), 'title'] = df['new_title']
df.drop('new_title', axis=1, inplace=True)

Merge the original df with titles from merges dataframe. After merge, overwrite the titles.

I hope this helps.