如何在Spark本身中使用Kafka在Spark流中实现偏移管理？

Question

I have to implement offset management within Spark for a streaming job in Java which reads from a Kafka stream. However, though the process has been described in the official documentation here , it does not actually give a code example of how to actually store and retrieve offsets from checkpoints. Rather, it cryptically says that

If you enable Spark checkpointing, offsets will be stored in the checkpoint.

Does this mean that if I just provide the checkpoint directory to the Spark context, it would automatically store offsets? What about retrieval of the last offset read when the application comes back on? The detail page on checkpointing that is linked there also leaves everything to the reader and only gives the syntax to set the checkpoint destination.

This and this give some clue as to how to use checkpoints, but in all of the instances, I can find that they have been used to cumulatively calculate something and not to store offsets. This question comes close, but still does not describe it.

Please help me in realizing this goal.

Answer 1

Saving Offsets in checkpoint does not work for you, because spark will save tasks in the check point so up-gradation of the code requires to delete the checkpoint. Instead you can save the offsets in Zookeeper,Kafka,File-System or any database.