使用 Jmx Exporter for Prometheus 缺少 Confluent Kafka Connect 指标

Question

I am not able to export "type=connector-metrics" metrics for Confluent connect service but other metrics are working fine.

I am using prometheus exporter java agent to expose metrics from Confluent connect as shown below.

Confluent Connect Configuration (/usr/bin/connect-distributed)

export KAFKA_OPTS='-javaagent:/opt/prometheus/jmx_prometheus_javaagent-0.12.0.jar=8093:/opt/prometheus/kafka-connect.yml'

kafka-connect.yml

   - pattern: kafka.connect<type=connector-metrics, connector=(.+)><>([a-z-]+)
     name: kafka_connector_$2
     labels:
       connector: "$1"
     help: "Kafka Connect JMX metric $1"
     type: GAUGE

With JMXTERM tool, i am able to see attributes for "type=connector-metrics"

$>bean kafka.connect:connector=local-file-sink,type=connector-metrics
#bean is set to kafka.connect:connector=local-file-sink,type=connector-metrics
$>info
#mbean = kafka.connect:connector=local-file-sink,type=connector-metrics
#class name = org.apache.kafka.common.metrics.JmxReporter$KafkaMbean
# attributes
  %0   - connector-class (double, r)
  %1   - connector-type (double, r)
  %2   - connector-version (double, r)
  %3   - status (double, r)
#there's no operations
#there's no notifications
$>get connector-type
#mbean = kafka.connect:connector=local-file-sink,type=connector-metrics:
connector-type = sink;

$>get status
#mbean = kafka.connect:connector=local-file-sink,type=connector-metrics:
status = running;

Confluent Connect Version - 5.4

Any suggestion to fix above issue?

Answer 1

I have been beating my head against the same issue. I have since found a fix by upgrading to the latest jmx_exporter (0.13.0) and used the example connector exporter config file ( https://github.com/prometheus/jmx_exporter ). I couldn't find a change log to tell what changed, but they "fixed" something.

Answer 2

Seems the problem is that the attribute type of connector-metric is double, but value is string.

Here example how I get status metric:

# kafka.connect:type=connector-metrics,connector="{connector}"
  - pattern: 'kafka.connect<type=connector-metrics, connector=(.+)><>status: ([a-z-]+)'
    name: kafka_connectors_status
    value: 1
    labels:
      connector: $1
      status: $2
    help: "Kafka Connect JMX metric connectors status"
    type: GAUGE