Prometheus 通过 JMX 监控 Apache Ignite

Question

I'm trying to monitor Apache Ignite with Prometheus' JMX exporter, but I'm seeing only default JVM metrics plus metrics only for "Thread Pools" Beans. JMX exporter run as agent:

/usr/bin/java -XX:+AggressiveOpts -javaagent:/etc/prometheus/jmx_prometheus_javaagent-0.13.0.jar=8080:/etc/prometheus/prometheus_config.yml -Xms1g -Xmx1g -server -XX:MaxMetaspaceSize=256m -Dfile.encoding=UTF-8 -Dcom.sun.management.jmxremote.rmi.port=49112 -Djava.rmi.server.hostname=127.0.0.1 -DIGNITE_QUIET=true -DIGNITE_SUCCESS_FILE=/usr/share/apache-ignite/work/ignite_success_ed3b2798-4d48-4188-94ac-1728fa8628dc -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=49112 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -DIGNITE_HOME=/usr/share/apache-ignite -DIGNITE_PROG_NAME=/usr/share/apache-ignite/bin/ignite.sh -cp /usr/share/apache-ignite/libs/*:/usr/share/apache-ignite/libs/ignite-indexing/*:/usr/share/apache-ignite/libs/ignite-spring/*:/usr/share/apache-ignite/libs/licenses/* org.apache.ignite.startup.cmdline.CommandLineStartup /etc/apache-ignite/default-config.xml

Ignite config has enables metrics:

<property name="metricExporterSpi">
    <list>
        <bean class="org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi"/>
    </list>
</property>

I can see a lot of ignite metrics from jconsole connected to 49112 port.

Tried different jmx-exporter options, nothing helped.

---
hostPort: 127.0.0.1:49112
lowercaseOutputLabelNames: true
lowercaseOutputName: true

---
lowercaseOutputLabelNames: true
lowercaseOutputName: true
rules:
- pattern: "^org.apache<clsLdr=(.+), name=sys"
  name: ignite_sys_stats
  help: Ignite cluster amount of heap memory in bytes
  labels:
    attr: $3
  type: GAUGE

and even empty config, that should mean "gather everything as is", still see just standard JVM + "Thread Pools". Can you suggest what's wrong here?

Answer 1

1. Create an empty prometheus_config.yml file. Make sure the file is properly read. Trace here: https://github.com/prometheus/jmx_exporter/blob/ce04b7dca8615d724d8f447fa25c44ae1c29238b/collector/src/main/java/io/prometheus/jmx/JmxCollector.java#L75 to make sure you are using the correct file.

2. remove the metricExporterSpi property

3. connect to the port specified, 8080 in your case, to see all the results.

Your pattern matching rules are not allowing Apache Ignite results to show properly.

You can use debug/tracing instructions here: https://github.com/prometheus/jmx_exporter to see what is happening.

pattern rule processing is here: https://github.com/prometheus/jmx_exporter/blob/ce04b7dca8615d724d8f447fa25c44ae1c29238b/collector/src/main/java/io/prometheus/jmx/JmxCollector.java#L357