[英]Kafka consumer lag monitoring visualization
I'm new to Kafka.我是卡夫卡的新手。 During study to kafka, I think monitoring consumer's lag is needed.
在学习kafka的过程中,我认为需要监控消费者的滞后。 When I search from google and docs, I found few ways.
当我从谷歌和文档搜索时,我发现了一些方法。
I just trying to get less pipeline steps.我只是想减少管道步骤。 What should be the simple way to visualize consumer's lag?
可视化消费者滞后的简单方法应该是什么?
If you are doing the setup in an organisation, datadog or prometheus is probably the way to go.如果您在组织中进行设置,datadog 或 prometheus 可能是 go 的方式。 You can capture other Kafka related metrics as well.
您也可以捕获其他与 Kafka 相关的指标。 These agents also have integrations with many other tools beside Kafka and will be a good common choice for monitoring.
这些代理还与 Kafka 之外的许多其他工具集成,将成为监控的一个很好的常见选择。
If you are just doing it for personal POC type of a project and you just want to view the lag, I find CMAK very useful ( https://github.com/yahoo/CMAK ).如果您只是为项目的个人 POC 类型而做,并且只想查看滞后,我发现 CMAK 非常有用( https://github.com/yahoo/CMAK )。 This does not have historical data, but provides a good current visual state of Kafka cluster including lag.
这没有历史数据,但提供了包括滞后在内的 Kafka 集群当前良好的可视化 state。
For cluster wide metrics you can use kafka_exporter ( https://github.com/danielqsj/kafka_exporter ) which exposes some very useful cluster metrics(including consumer lag) and is easy to integrate with prometheus and visualize using grafana.对于集群范围的指标,您可以使用 kafka_exporter ( https://github.com/danielqsj/kafka_exporter ),它公开了一些非常有用的集群指标(包括消费者滞后),并且易于与 prometheus 集成并使用 grafana 进行可视化。
Burrow is extremely effective and specialised in monitoring consumer lag.Burrow is good at caliberating consumer offset and more importantly validate if the lag is malicious or not. Burrow 非常有效,专门用于监控消费者滞后。Burrow 擅长校准消费者偏移,更重要的是验证滞后是否是恶意的。 It has integrations with pagerduty so that the alerts are pushed to the necessary parties.
它与 pagerduty 集成,以便将警报推送给必要的各方。
https://community.cloudera.com/t5/Community-Articles/Monitoring-Kafka-with-Burrow-Part-1/ta-p/245987 https://community.cloudera.com/t5/Community-Articles/Monitoring-Kafka-with-Burrow-Part-1/ta-p/245987
What burrow has:洞穴有什么:
If you are looking for quick solution you can deploy burrow followed by the burrow front end https://github.com/GeneralMills/BurrowUI如果您正在寻找快速解决方案,您可以部署 burrow,然后部署 burrow 前端https://github.com/GeneralMills/BurrowUI
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.