[英]How to monitor consumer lag in kafka via jmx?
I have a kafka setup that includes a jmx exporter to prometheus. 我有一个kafka设置,包括prometheus的jmx导出器。 I'm looking for a metric, that gives the offset lag based on topic and groupid.
我正在寻找一个指标,根据主题和groupid给出偏移滞后。 I'm running kafka 2.2.0.
我正在运行kafka 2.2.0。
Some resources online point to a metric called kafka.consumer
, but I have no such metric in my setup. 一些在线资源指向一个名为
kafka.consumer
的指标,但我的设置中没有这样的指标。
From my jmxterminal: 从我的jmxterminal:
$>domains
#following domains are available
JMImplementation
com.sun.management
java.lang
java.nio
java.util.logging
jdk.management.jfr
kafka
kafka.cluster
kafka.controller
kafka.coordinator.group
kafka.coordinator.transaction
kafka.log
kafka.network
kafka.server
kafka.utils
I am, however, able to see the data I need by using the following command: 但是,我可以使用以下命令查看我需要的数据:
root@kafka-0:/kafka# bin/kafka-consumer-groups.sh --describe --group benchmark_consumer_group --bootstrap-server localhost:9092
Consumer group 'benchmark_consumer_group' has no active members.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
benchmark_topic_10B 2 2795128 54223220 51428092 - - -
benchmark_topic_10B 9 4 4 0 - - -
benchmark_topic_10B 6 7 7 0 - - -
benchmark_topic_10B 7 5 5 0 - - -
benchmark_topic_10B 0 2834028 54224939 51390911 - - -
benchmark_topic_10B 1 15342331 54222342 38880011 - - -
benchmark_topic_10B 4 5 5 0 - - -
benchmark_topic_10B 5 6 6 0 - - -
benchmark_topic_10B 8 8 8 0 - - -
benchmark_topic_10B 3 4 4 0 - - -
But that does not help since I need to track if from a metric. 但这无济于事,因为我需要从指标中跟踪。 Also, this command takes about 25 seconds to execute, making it unreasonable to use as a source for metrics.
此外,此命令执行大约需要25秒,因此将其用作指标的来源是不合理的。
My guess is that the metric kafka.consumer
does not exist in version 2.2.0 and was replaced with another. 我的猜测是,版本2.2.0中不存在度量标准
kafka.consumer
,而是替换为另一个。 Although, I can't find any resources online with up-to-date information on how and where to get that metric 虽然,我在网上找不到任何有关如何以及在何处获得该指标的最新信息的资源
The kafka.consumer
JMX metrics are only present on the consumer processes themselves, not on the Kafka broker processes. kafka.consumer
JMX指标仅出现在消费者流程本身,而不是Kafka经纪人流程上。 Note that you would not get the kafka.consumer
metric from consumers using a consumer library other than the Java one. 请注意,您不会使用除Java之外的使用者库从消费者那里获得
kafka.consumer
指标。
Currently, there are no available JMX metrics for consumer lag from the Kafka broker itself. 目前,Kafka经纪商本身没有可用的消费者滞后JMX指标。 There are other solutions that are commonly used for monitoring consumer lag, such as Burrow by LinkedIn.
还有其他常用于监控消费者滞后的解决方案,例如LinkedIn的Burrow 。 There are also a few open source projects such as kafka9.offsets that expose consumer lag metrics via JMX, but may not be updated to work with the latest Kafka.
还有一些开源项目,例如kafka9.offsets ,它们通过JMX公开消费者滞后指标,但可能无法更新以使用最新的Kafka。
You can give Kafka Minion ( https://github.com/cloudworkz/kafka-minion ) a try. 您可以试试Kafka Minion( https://github.com/cloudworkz/kafka-minion )。 While Kafka Minion internally works similiarly as Burrow (consumes __consumer_offsets topic for Consumer Group Offsets) it has several advantages for your use case
虽然Kafka Minion内部的工作方式与Burrow类似(消费群组偏移消耗__consumer_offsets主题),但它对您的用例有几个优点
Advantages of Kafka Minion over Burrow for your case: Kafka Minion优于Burrow的优势:
Disclaimer: I am the author of Kafka Minion, and I am still looking for more feedback from other users. 免责声明:我是Kafka Minion的作者,我仍在寻找其他用户的更多反馈。 I intend to actively maintain and develop the exporter for my projects, the company I am working for and for the community.
我打算为我的项目,我为公司工作的公司和社区积极维护和发展出口商。
To answer your question regarding what you are seeing using the kafka-consumer-groups.sh
shell script. 使用
kafka-consumer-groups.sh
shell脚本回答有关您所看到的问题。 This won't work as it cannot report lags for inactive consumers which is a bit counterproductive. 这不起作用,因为它无法报告非活动消费者的滞后,这有点适得其反。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.