比较两个数据帧以查找Spark中的子字符串

Question

I have three dataframes, dictionary,SourceDictionary and MappedDictionary. The dictionary andSourceDictionary have only one column, say words as String. The dictionary which has million records, is a subset of MappedDictionary (Around 10M records) and each record in MappedDictionary is substring of dictionary. So, I need to map the ditionary with SourceDictionary to MappedDictionary. Example:

Records in ditionary : BananaFruit, AppleGreen
Records in SourceDictionary : Banana,grape,orange,lemon,Apple,...

Records to be mapped in MappedDictionary (Contains two columns) :

BananaFruit Banana
AppleGreen Apple

I planned to do like two for loops in java and make substring operation but the problem is 1 million * 10 million = 10 Trillion iterations Also, I can't get correct way to iterate a dataframe like a for loop Can someone give a solution for a way to make iteration in Dataframe and perform substring operations? Sorry for my poor English, I am a non-native Thanks for stackoverflow community members in advance :-)

Answer 1

Though you have million record in sourceDictionary because it has only one column broadcasting it to every node won't take up much memory and it will speed up the total performance.

import org.apache.spark.broadcast.Broadcast
import org.apache.spark.sql.catalyst.encoders.RowEncoder

//Assuming the schema names
val SourceDictionarySchema = StructType(StructField("value",StringType,false)) 
val dictionarySchema = StructType(StructField("value",StringType,false))
val MappedDictionary = StructType(StructField("value",StringType,false), StructField("key",StringType,false))

val sourceDictionaryBC = sc.broadcast(
   sourceDictionary.map(row =>
      row.getAs[String]("value")
   ).collect.toList
)

val MappedDictionaryN = dictionary.map(row =>
   val value = row.getAs[String]("value")
   val matchedKey = sourceDictionaryBC.value.find(value.contains)

   Seq(value, matchedKey.orNull)
)(RowEncoder(MappedDictionary))

After this you have all the new mapped records. If you want to combine it with the existing MappedDictionary just do a simple union.

MappedDictionaryN.union(MappedDictionary)