火花结构化流的 LSHModel

Question

Apparently, the LSHModel of MLLib from spark 2.4 supports Spark Structured Streaming ( https://issues.apache.org/jira/browse/SPARK-24465 ).

However, it's not clear to me how. For instance an approxSimilarityJoin from MinHashLSH transformation (https://spark.apache.org/docs/latest/ml-features#lsh-operations ) could be applied directly to a streaming dataframe?

I don't find more information online about it. Could someone help me?

Answer 1

You need to

1. Persist the trained model (eg modelFitted ) somewhere accessible to your Streaming job. This is done outside of your streaming job.

modelFitted.write.overwrite().save("/path/to/model/location")

2. Then load this model within you Structured Streaming job

import org.apache.spark.ml._
val model = PipelineModel.read.load("/path/to/model/location")

3. Apply this model to your streaming Dataframe (eg df ) with

model.transform(df)

// in your case you may work with two streaming Dataframes to apply `approxSimilarityJoin`.

It might be required to get the streaming Dataframe into the correct format to be used in the model prediction.