Kafka 消费者通过 JMX 滞后

Question

I'm trying to monitor the lag of a consumer group in Kafka 0.10.

Our consumers are keeping track of their offsets in Kafka rather than ZooKeper. This means I can get the figures using the following:

bin/kafka-consumer-groups.sh --bootstrap-server <broker> --describe --group <group-name>

This works fine. However, my broker already makes use of the Prometheus JMX exporter for collecting a number of stats. I've connected JConsole to the brokers but can't see the same data being reported in JMX as reported by kafka-consumer-groups.sh .

Is there anyway to get this information from Kafka with JMX without needing any additional tools?

Answer 1

You could retrieve the atrributes {topic}-{partition}.records-lag of metric kafka.consumer:type=consumer-fetch-manager-metrics,client-id={client-id} for all partitions. That should be equivalent to the output of consumer-groups.sh

Answer 2

To expose JMX metrics from consumer simply add

-Dcom.sun.management.jmxremote.port=9090 
-Dcom.sun.management.jmxremote.authenticate=false  
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=false

parameters to your java based consumer application, then you can access it using jconsole tool and localhost:9090 url. There you can find kafka.consumer:type=consumer-fetch-manager-metrics bean.