flink 是否为数据集批处理提供检查点

Question

How to configure check pointing for flink batch processing. I'm interested in knowing how checking pointing work internally. Since check point happens at an interval, if the job failed before the next point, won't there be any duplicate processing if it restarts. Does flink check points for each operator, sink and sources?

Answer 1

Flink does not support checkpointing on the DataSet API.

You can use checkpointing in DataStream with finite sources though, which covers most of the DataSet API use cases already. The long-term vision is to completely replace the DataSet API with DataStream + finite sources, such that users do not need to write two programs if they want to analyze a stream or batch.

With Table API and SQL, this goal is already pretty near.