计算 R 中两个向量/字符串之间的相似度

Question

It might be similar question asked in this forum but I feel my requirement peculiar. I have a data frame df1 where it consists of variable "WrittenTerms" with 40,000 observations and I have another data-fame df2 with variable "SuggestedTerms" with 17,000 observations

I need to calculate the similarity between "written Term" and "suggestedterms"

df1$WrittenTerms

head pain

lung cancer

abdminal pain

df2$suggestedterms

cardio attack

breast cancer

abdomen pain

head ache

lung cancer

I need to get the output as follow

df1$WrittenTerms df2$suggestedterms Similarity_percentage

head pain head ache 50%

lung cancer lung cancer 100%

abdminal pain abdomen pain 80%

I am writing the below code to meet the requirement but its taking more time as it involves for loop and is there any way where we can find similarity using TF IDF OR any other approach which will take less time

df_list <- data.frame(check.names = FALSE) # Creating empty dataframe

# calculating similarity between strings.

for(i in df1$WrittenTerms){
  df2$oldsim<- stringdist(i,df2$suggestedterms,method = "lv")
  df2$oldsim <- 1 - df2$oldsim / nchar(as.character(df2$suggestedterms))
  df2 <- head(df2[order(df2$oldsim, decreasing = TRUE),],1)
  df_list <- rbind(df_list, df2)
}

df1 <- cbind(df1, df_list)

Answer 1

The base library's adist function gives you Levenshtein distances between two arrays, returning a matrix of distances for each pair of entries. You could write a function that converts the Levenshtein metric into your transformation:

my_dist <- function(x, y) 1 - adist(x, y) / nchar(y)
x <- my_dist(df1$WrittenTerms, df2$suggestedterms)

Now obtain the maximum value of your metric for each row of x, which will be the best suggestedterm for each WrittenTerms :

mx <- apply(x, 1, function(y) {mx <- which.max(y); c(y[mx], mx)})

Your final desired data frame could then be constructed as follows:

data.frame(Written.Terms = df1$WrittenTerms, 
           suggestedterms = df2$suggestedterms[mx[2, ]], 
           Similarity_percentage = mx[1, ])