In my app I will perform some kind of health check of my Kafka cluster.
Currently I make a TopicMetadataRequest
to detect dead brokers:
Future {
// this will fail if Kafka is unavailable
consumer.send(new TopicMetadataRequest(Seq("health-check-topic"), 1))
}
Unfortunately this call produces a huge network traffic, because of Cluster topology/settings.
Is there a better way to check kafka brokers? What I need is something simple like true/false
indicator.
If you want to build your own health check, this is a current (January 2020) list of KIPs covering health checks:
Regarding Harvinder Singh 's currently accepted answer :
Kafka Manager is great but it's evolving slowly . There's of course Confluent Control Center - a part of Confluent Platform, but you'll need a license for it. Confluent is a company founded by the team that built Apache Kafka. I've heard about akHQ (ex KafkaHQ) ( HackerNews story ). Here's a list of management consoles maintained on Apache Kafka Confluence page (check URLs there):
If you don't need GUI, there are also:
I would strongly recommend you to use Yahoo Kafka Manager, which provides all the information related to Kafka setup. (eg bytes sent/consumed over a time interval). This tool can also be used for managing your Kafka Cluster.
It also exposes Restful API and you can consume these API in your own application, if needed. Follow the following link to access it.
You can also use Zookeeper API to get the broker list as follows:
ZooKeeper zk = new ZooKeeper(KafkaContextLookupUtil.getZookeeperConnect().getZkConnect(), 10000, null);
List<String> ids = zk.getChildren("/brokers/ids", false);
List<Map> brokerList = new ArrayList<>();
ObjectMapper objectMapper = new ObjectMapper();
for (String id : ids) {
Map map = objectMapper.readValue(zk.getData("/brokers/ids/" + id, false, null), Map.class);
brokerList.add(map);
}
return brokerList;
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.