使用 State (Pyspark) 的 Spark 结构化流

Question

I want to match data with spark streaming based on a certain condition and I want to write this data to Kafka. By keeping the unmatched under a state and this state will keep a maximum of 2 days of data in hdfs. Each new incoming data will try to match the unmatched data in this state. How can I use this state event? (I'm using pyspark)

Answer 1

Pyspark doesn't support stateful implementation by default .

Only Scala/Java API has this option using mapGroupsWithState function on KeyValueGroupedDataSet

But you can store 2 days of data in somewhere else ( file system or some no sql database ) and then for each new incoming data you can go to nosql database and fetch corresponding data and do the remaining stuff.