如何在Spark Streaming应用程序中初始化DStream的状态

Question

I have a spark-streaming application that basically keeps track of a string->string dictionary.

So I have messages coming in with updates, like:

“A”->”B”

And I need to update the dictionary.

This seems like a simple use case for the updateStateByKey method.

However, my issue is that when the app starts I need to “initialize” the dictionary with data from a hive table, that has all the historical key/values with the dictionary.

The only way I could think of is doing something like:

val rdd =… //get data from hive
def process(input: DStream[(String, String)]) = {
    input.join(rdd).updateStateByKey(update)
}

So the join operation will be done on every incoming buffer, where in fact I only need it on initialization.

Any idea how to achieve that?

Thanks

Answer 1

PairDStreamFunctions.updateStateByKey有一个过载，它接受一个I​​nitialRDD ，这似乎是您需要的：

updateStateByKey[S](updateFunc: (Seq[V], Option[S]) ⇒ Option[S], partitioner: Partitioner, initialRDD: RDD[(K, S)])(implicit arg0: ClassTag[S]): DStream[(K, S)]