kafka（非活动）消费者滞后监控

Question

I using Confluent Kafka 0.10.0 . I want to monitor consumer offset(s) for purposes of alerting.

Where are the values for a Consumer Group stored when the group isn't active? Is there a programmatic method to extract these? (IE something in the kafka api)

Similarly - is there a way to get a list of groups that includes the inactive ones?

---

Note:

The only way to (presently) list / describe the groups for my project is to use
kafka-consumer-groups ... --new-consumer .

Answer 1

Do the following things:

1. Get your group.id, and calculate Math.abs(groupID.hashCode()) % 50. Say your group.id is "test_group1", then get the target partition number by calculating Math.abs("test_group1".hashCode()) % 50

2. Run the command below using the target partition number calculated previously:

   bin/kafka-simple-consumer-shell.sh --topic __consumer_offsets --partition --broker-list *** --formatter "kafka.coordinator.GroupMetadataManager\\$OffsetsMessageFormatter"

You should be able to see the offsets for this inactive group. For active groups, just running kafka-consumer-groups script is enough.

Answer 2

Have a look at the kafka documentation. The following page claims to be a working example of just what you want: https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka