Kafka on Kubernetes using Strimzi : monitoring/exposing all JMX metrics to prometheus

Question

I am running kafka on Kubernetes using the Strimzi operator. For monitoring I am using Prometheus and I followed the installation guide as described in Strimzi deployment guide.

https://strimzi.io/docs/operators/master/deploying.html

For the Kafka cluster deployment I used the following template from strimzi repository. https://github.com/strimzi/strimzi-kafka-operator/blob/master/examples/metrics/kafka-metrics.yaml . The metrics to be exposed to Prometheus are defined in a configmap in the same file.

My question is that if I want Prometheus to scrape all the JMX metrics from Kafka ; Is there any ready JMX configuration file to pass to the configmap instead of the current one. Please note that I am particularly interested in consumer group metrics (rebalance-latency-avg, rebalance-rate-per-hour, etc…).

Does monitoring kafka through JMX/prometheus cause any performance degradation to Kafka?

Answer 1

When you define the JMX Exporter configuration file as empty, for example something like this:

kind: ConfigMap
apiVersion: v1
metadata:
  name: kafka-metrics
  labels:
    app: strimzi
data:
  kafka-metrics-config.yml: |
    lowercaseOutputName: true

It will export all metrics in the raw format. This allows you see all of them. However, they will look quite different from what you get with the Strimzi example because they will not have the rules which translate them, extract some parts as labels etc.

So in most cases you will want to add these to the rules => you have to define pattern to match the metric you are interested in, define what type it should be and how it should translate into the metric name and the labels. This can be just added to the existing rules in the Strimzi example.