如何在 Apache Kafka-Connect 服务器中重置源连接器偏移量

Question

After deleting the source connector from Kafka connect, saved offsets for this connector are still present in the Kafka. When the same connector with the same name is started again then it starts to process data from the last saved offset. Changing the name of the connector is not a solution since it makes hard to automate connector submission.

Is there a workaround for this problem (renaming the source connector is not an options)

Answer 1

Consume the offsets topic used by Connect; it is JSON, so you can inspect its format.

Construct a series of messages that set the offset to 0 for the source

For a full example using the FileStreamSource, see this post

There is a KIP-199 and JIRAs as well that have been asking for tooling around this, also asking that deleting a source connector should delete its offsets.

Answer 2

what I found is that if you know the topic that the Kafka Connector is using, this might work: Kafka : Reset offset of a specific partition of topic and here's another reference that might help:https://gist.github.com/marwei/cd40657c481f94ebe273ecc16601674b

I think I have had a similar issue and what I did was delete the topics that started with offset, which meant that I had to stop Kafka Connect, delete the applicable offset topics, and then restart Kafka Connect. Not sure if that's the best option, but here are some options for your request.

Answer 3

Reset the offset for the consumer group for the kafka-connect connector. The consumer group for all connectors are different - so it should be okay and not affect any other connectors.