模糊搜索 pyspark dataframe

Question

I have a large csv file (>96 million rows) and seven columns. I want to do a fuzzy search on one of the columns and retrieve the records with the highest similarity to the input string. The file is managed by spark and I load it via pyspark into some dataframe. Now I want to use something like fuzzywuzzy to extract the rows matching the best.

But the fuzzywuzzy function extract returns something that I cannot work with:

process.extract("appel", df.select(df['lowercase']), limit=10)

Result: [(Column<'lowercase'>, 44)]

df is the pyspark dataframe (load using spark.read.csv), the column I want to search on is 'lowercase' and I want to retrieve all other columns for the respective rows plus the similarity score.

Any suggestions?

Answer 1

You can try using other python libraries like Rapidfuzz , which computes fuzzy string match taking an input string and list of strings, as input. You can use choose your desired string match algorithm to compute appropriate matches.

The code would look something like this:

# pip install rapidfuzz
from rapidfuzz import fuzz, process

input_string = 'example string'
query_list = df.select(df['lowercase'])

results = process.extract(input_string, query_list, scorer=fuzz.token_ratio, limit=1)

# output format = [('string1', confidence_score, index_in_list), ...]

Answer 2

from pyspark.sql.functions import udf
from pyspark.sql.types import StringType
from fuzzywuzzy import fuzz
    
    
def match_string(s1, s2):
    val = fuzz.token_sort_ratio(s1, s2)
    return val
    
MatchUDF = udf(match_string, StringType())
    
scores_df = df.withColumn("similarity_score", MatchUDF(F.col("name_1"), F.col("name_2")))\
            .withColumn("run_date", F.current_date())
    
scores_df.show()