如果我在kafka流中设置commit.interval.ms =的值，是否可以提交偏移量？

Question

I am trying to understand behavior of kafka streams related to offset commit. Streams commit offset manually as auto commit is set to " false " in Streams. What if I keep commit.interval.ms = 0 , whether streams will work correctly?

Answer 1

What are you trying to accomplish by setting that config parameter to zero milliseconds? The default is 100 milliseconds for Streams, per the official docs:

Note, if 'processing.guarantee' is set to 'exactly_once', the default value is 100, otherwise the default value is 30000.

From the official Kafka Streams docs - https://kafka.apache.org/documentation/#streamsconfigs

Updated:

Based on your comment, I believe that yes, you should be able to store commits in an external database, and that commit interval configuration would still apply.

Also, make sure to go over the docs on the implications of storing offsets outside of Kafka - https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#rebalancecallback

Answer 2

If you set commit.interval.ms = 0 Kafka Streams will commit "as soon as possible". In the implementation, there is a poll-process-loop that checks if a commit is required. If you set commit.interval.ms = 0 this check will evaluate to true every time and thus commit will happen each time.

When the commit condition is checked is an internal implementation detail and there is no public contract how ofter the condition is checked. Note, it's not recommended to commit too frequently because it put additional load on Kafka Streams client and the brokers because committing is a sync operation and not for free.

From the comment:

We want to check whether we can store commits in external db..so we want to know can we achieve offset commit management solely on db side

Using Streams API, you cannot store offsets in an external DB. Kafka Streams uses Kafka's group management and commit API and you cannot change this.