Spark结构化流自定义StateStoreProvide

Question

By default Structured Streaming job is using HDFSStateStoreProvide . The issue with using HDFS store is that it is not scalable. When the job get more data from the kafka during high traffic hours, it fails due following error:

18/12/06 15:54:35 ERROR scheduler.TaskSetManager: Task 191 in stage 231.0 failed 4 times; aborting job
18/12/06 15:54:35 ERROR streaming.StreamExecution: Query eventQuery [id = 42051afe-b1bc-438d-8143-2d7e5def717c, runId = 6201c769-b115-4b92-bad5-450b8803b88b] terminated with error
org.apache.spark.SparkException: Job aborted due to stage failure: Task 191 in stage 231.0 failed 4 times, most recent failure: Lost task 191.3 in stage 231.0 (TID 24016, sparkstreamingc1n5.host.bo1.csnzoo.com, executor 659): java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider.org$apache$spark$sql$execution$streaming$state$HDFSBackedStateStoreProvider$$readSnapshotFile(HDFSBackedStateStoreProvider.scala:481)
    at org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider$$anonfun$org$apache$spark$sql$execution$streaming$state$HDFSBackedStateStoreProvider$$loadMap$1.apply(HDFSBackedStateStoreProvider.scala:359)
    at org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider$$anonfun$org$apache$spark$sql$execution$streaming$state$HDFSBackedStateStoreProvider$$loadMap$1.apply(HDFSBackedStateStoreProvider.scala:358)
    at scala.Option.getOrElse(Option.scala:121)

How to configure a custom state store provide?

For testing purposes I tried adding a fake class

--conf spark.sql.streaming.stateStore.providerClass=com.streaming.state.RocksDBStateStoreProvider

But the job is still picking the HDFSStateStoreProvider even when this class doesn't exist. Is this a expected behavior?

Can I use any key value database to write the custom state provider?

Or it is only limited to RocksDB and Cassandra .

Answer 1

How to configure a custom state store provide?

Your approach for configuring custom state store provider looks correct, but you can't change state store provider once you ran query before. (Spark will read the configuration from metadata in checkpoint.) This restriction makes sense since when changing state store provider, state is not guaranteed to be restored.

Can I use any key value database to write the custom state provider?

There's no specific restriction, once your custom state provider implements spec on state store provider. Two major things to consider are 1. Spark will checkpoint the change for every batch 2. Spark requires state store provider to restore state at specific version. Your custom state provider should be performant - because it will add latency on every batch.

You may also want to consider (transitive) dependency being added to Spark application due to custom state provider.