Check if kafka cluster is operating healthy

Question

As a consultant helping Cx's with Apache Kafka deployments I want to check if we have any open source automated script that can run against Kafka cluster and pull detailed report and check if cluster has any bad configuration or cluster is not following standard best practices? Below are few things script can check

• Is cluster overloaded?
• How many topics and partitions in cluster?
• Does cluster has right numbers of topics based on allocated infrastructure?
• Is any topic going through throughput skew?
• is cluster being overloaded with too many client connections?
• Is cluster facing any performance issues?

Answer 1

cluster overloaded?

Setup generic CPU, network + JVM monitoring

How many topics and partitions in cluster?

I believe JMX can check this, but most existing Kafka GUI tools have this as well

right numbers of topics based on allocated infrastructure?

There is no "right number". Latest versions of Kafka can reportedly have tens to hundreds of thousands of topics, even on a few brokers

topic going through throughput skew?

In my experience, the only reasonable way to detect this is to use Producer interceptors collecting what partitions are used, or have SSH access to the brokers and inspecting the log segment data

overloaded with too many client connections? ... facing any performance issues?

See first comment.