在Spark结构化流媒体中使用Kafka接收器时，检查点是否必须执行？

Question

I'm trying to use Spark Structured Streaming to write aggregated data to Kafka. Here's my code:

dataset
    .writeStream()
    .queryName(queryName)
    .outputMode(OutputMode.Append())
    .format("kafka")
    .option("kafka.bootstrap.servers", kafkaBootstrapServers)
    .option("topic", "topic")
    .trigger(Trigger.ProcessingTime("15 seconds"))
    // .option("checkpointLocation", checkpointLocation)
    .start();

If I comment out checkpointLocation , I get:

Exception in thread "main" org.apache.spark.sql.AnalysisException: checkpointLocation must be specified either through option("checkpointLocation", ...) or SparkSession.conf.set("spark.sql.streaming.checkpointLocation", ...);
    at org.apache.spark.sql.streaming.StreamingQueryManager$$anonfun$3.apply(StreamingQueryManager.scala:210)
    at org.apache.spark.sql.streaming.StreamingQueryManager$$anonfun$3.apply(StreamingQueryManager.scala:205)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.sql.streaming.StreamingQueryManager.createQuery(StreamingQueryManager.scala:204)
    at org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala:278)
    at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:282)
    at <myClass>)

Is checkpointing mandatory when using a Kafka sink? I could not find an answer in the documentation.

Answer 1

The checkpointing is needed to keep track what exactly was processed and written to a sink.

Let's assume you have a bunch of files in an input folder. When you start a stream spark starts processing files from source. To be sure that these files are processed and written to a sink only once it uses checkpointing where all progress information is stored.

In other words, checkpointing is needed not for the sink but for the entire stream to make sure that the same input data won't be processed over and over again.