kafka multiple consumer groups

Question

I have multiple consumer groups each having 1 consumer.

Say, Topic T1 and Partition P1 and 2 consumer group CG1 and CG2 CG1 has 1 consumer - CG1#C1 and correspondingly CG2 has CG2#C1

Now we are receiving the same message across multiple consumer groups which is expected as per design.

My question is how Kafka maintains the offset across multiple consumer groups. If 1 consumer group is down for few minutes and agian it is backed up then how it will get the message from the last committed read. CG1 is in offset 24 and CG2' before going down committed till offset 10. so when CG2 is up how it will start getting from offset 11 and CG1 is not impacted.

Answer 1

By default the offsets are stored in a special topic called __consumer_offsets . This is a compacted topic , where the offsets are stored as messages with a key which contains the topic, partition and the consumer group as a key. Thanks to this the offsets for different consumer groups are independent. Hope this answers your question. For more information about how exactly this is handled, you can have a look here: http://kafka.apache.org/documentation/#impl_offsettracking

Answer 2

Consumer groups are independent entities. If consumers in the different consumer groups, subscribe to the same kafka topic, there won't be any clash between offset management. Each consumer group will maintain it's own offset in __consumer_offsets topics(this is the compacted internal topic managing the consumer offsets)

Kafka provides the option to store all the offsets for a given consumer group in a designated broker (for that group) called the offset manager. Whenever a consumer reads a message from a partition, it sends the offset commit request to offset manager. Once the commit request is accepted, the consumer can read the next offset.

In case of offset commit failure, consumer retries to commit. Each offset commit is maintained per partition.

Here is the detailed explanation : http://kafka.apache.org/documentation/#impl_offsettracking