Micrometer metrics & Application Insights java agent - Metrics accumulate although they shouldn't (or am I missing something)

Question

We have a Java EE JAX-RS application (Java EE 8 hosted on WildFly 20), in which we have configured Micrometer (v1.6.4) as a metrics framework. We are trying to connect it with Azure Application Insights so that metrics are made available to the relevant Application Insights resource (and then add alerts etc.).

In order to do so, we have attached the codeless Application Insights agent (v3.0.2), which automatically publishes Micrometer metrics without any code changes. On top of that, we have added a PrometheusMeterRegistry in the globalRegistry, only for debugging purposes (we do not use Prometheus).

Metrics.globalRegistry.add(new PrometheusMeterRegistry(PrometheusConfig.DEFAULT));

On top of that, we have configured an endpoint which returns the Prometheus data using the following snippet:

PrometheusMeterRegistry promRegistry = (PrometheusMeterRegistry) registry;
result = promRegistry.scrape();

The agent has a configuration file attached to it in which only the connectionString and role name of the application are set.

The agent starts correctly, and data are published in the Application Insights resource. Custom metrics are available for selection in the Monitoring -> Metrics area of the resource.

The issue we are facing is that the metrics data accumulate over time although they shouldn't. And to give an example:

• We have a Timer setup in order to measure time of a specific operation in our application. The Timer is triggered by an @Interceptor using the @AroundInvoke method, and records the time take for the actual operation to complete. We use this code snippet to record the time

...
Builder timerBuilder = Timer.builder(timerName);
Timer timer = timerBuilder.register(metricsProducer.getMetricsRegistry());

return timer.record(() -> {
    try {
        return ctx.proceed();
    } catch (Exception e) {
        throw new RuntimeException(e.getMessage());
    }
});

• We have a Counter also setup which is used to measure the number of times that a specific operation has been called (I know that the same can be achieved through the Timer, but it's only there for testing purposes)

Assume that we call the operation that is bound to the Counter only once. We expect that the Counter value is 1, which is something that we confirm with logging. Assume also that we call the operation that is bound to the Timer only once. We expect that only this triggering gets measured and recorded.

In the Application Insights resource, we see that the metric information gets updated every time the agent publishes information, and modifies the values without though any trigger in the backend. In particular, both of the metrics increase the Count aggregated value, but since for the Timer the duration does not change, the Average falls (suggesting that a metric value is passed with duration 0).

On the other hand, when calling the Prometheus scraping endpoint, data are returned correctly, ie Count for both of the metrics stays to 1 and data are the expected ones.

I have tried the same using SpringBoot as the underlying framework, again with the same results.

Any thoughts? Am I missing something?

Thank you

Vangelis

Answer 1

It's an issue in micrometer library, and a relevant PR has been created

Answer 2

Is it possible that this issue in Micrometer is related to what I describe here ?