在Java代码中检查Kafka消费者组的活跃性

Question

I have written a java code for Kafka consumer using executor service which will run 24/7 consume the data and process it.

But in some scenarios like due to Kafka coordinator broker node went down, job will fail ie, consumer group will become dead and i will not able to know it and the java jar would be running but not doing anything.

 INFO [2018-11-07 03:25:58,425] [pool-1-thread-1] (AbstractCoordinator.java:coordinatorDead) - Marking the coordinator broker_host (id: 2147483446 rack: null) dead for group group_name
 INFO [2018-11-07 03:26:10,817] [pool-1-thread-1] (AbstractCoordinator.java:onSuccess) - Discovered coordinator broker_host (id: 2147483446 rack: null) for group group_name.

I want to check the Kafka consumer liveliness in java code periodically. So, if Kafka consumer group becomes dead, i will be able to make it live or trigger the mail and kill the job.

Can anyone provide any input on this.

Answer 1

If the broker acting as coordinator for your consumer group goes down, each consumer in the group will find which remaining broker is to become the new coordinator for their group. This scenario should not be fatal to your group.

However, monitoring consumer groups is still a very common thing to do in order to track lag, scale in/out if needed or just ensure an important job is actually running!

The easiest way is to use the AdminClient APIs, especially:

• describeConsumerGroups() : This lists the members and the overal state of the group
• listConsumerGroupOffsets() : This lists the current offsets

Alternatively you can run the kafka-consumer-groups.sh tool to retrieve details about groups. That displays the current position, lag and client ids or members.