Pandas : if value in a dataframe contains string from another dataframe, append columns

Question

Let's say I have two dataframes df1 and df2. I want to append some columns of df2 to df1 if the value of a specific column of df1 contains the string in a specific column of df2, NaN if not.

A small example:

import pandas as pd
df1 = pd.DataFrame({'col': ['abc', 'def', 'abg', 'xyz']})
df2 = pd.DataFrame({'col1': ['ab', 'ef'], 'col2': ['match1', 'match2'], 'col3': [1, 2]})

df1:
   col
0  abc
1  def
2  abg
3  xyz

df2:

  col1    col2    col3
0   ab  match1       1
1   ef  match2       2

I want:

   col   col2_match   col3_match
0  abc       match1            1
1  def       match2            2
2  abg       match1            1
3  xyz          NaN          NaN

I managed to do it in a dirty and unefficient way, but in my case df1 contains like 100K rows and it takes forever...

Thanks in advance !

EDIT

A bit dirty but gets the work done relatively quickly (I still thinks there exists a smartest way though...):

import pandas as pd
import numpy as np


df1 = pd.DataFrame({'col': ['abc', 'def', 'abg']})
df2 = pd.DataFrame({'col1': ['ab', 'ef'],
                    'col2': ['match1', 'match2'],
                    'col3': [1, 2]})


def return_nan(tup):
    return(np.nan if len(tup[0]) == 0 else tup[0][0])


def get_indexes_match(l1, l2):
    return([return_nan(np.where([x in e for x in l2])) for e in l1])


def merge(df1, df2, left_on, right_on):
    df1.loc[:, 'idx'] = get_indexes_match(df1[left_on].values,
                                          df2[right_on].values)
    df2.loc[:, 'idx'] = np.arange(len(df2))
    return(pd.merge(df1, df2, how='left', on='idx'))


merge(df1, df2, left_on='col', right_on='col1')

Answer 1

You can use python difflib module for fuzzy match like this

import difflib 
difflib.get_close_matches
df1.col = df1.col.map(lambda x: difflib.get_close_matches(x, df2.col1)[0])

So now your df1 is

    col
0   ab
1   ef
2   ab

You can call it df3 if you wish to keep df1 unaltered.

Now you can merge

merged = df1.merge(df2, left_on = 'col', right_on = 'col1', how = 'outer').drop('col1', axis = 1)

The merged dataframe looks like

    col col2    col3
0   ab  match1  1
1   ab  match1  1
2   ef  match2  2

EDIT: In case of no match like the new example given, you just need to put a conditional in lambda

df1.col = df1.col.map(lambda x: difflib.get_close_matches(x, df2.col1)[0] if difflib.get_close_matches(x, df2.col1) else x)

Now after the merge you get

    col col2    col3
0   ab  match1  1
1   ab  match1  1
2   ef  match2  2
3   xyz NaN     NaN