How to define additional or custom tags for default Spring Boot 2 metrics?

Question

Recently I switched to Spring Boot 2 with Micrometer . As I got these shiny new metrics, I talked with our DevOps guys and we started exporting them to Telegraf .

To distinguish between different applications and application nodes, we decided to use tags. This works perfectly for custom metrics, but now I started thinking about the pre-defined. To achieve the same for default metrics, I need the ability to add extra tags for them as well.

Is it possible to achieve this? Am I doing this right?

EDIT: I tried next approach:

@Component
public class MyMetricsImpl implements MyMetrics {

    @Autowired
    protected MyProperties myProperties;
    @Autowired
    protected MeterRegistry meterRegistry;

    @PostConstruct
    public void initialize() {
        this.meterRegistry.config()
                .commonTags(commonTags());
    }

    @Override
    public List<Tag> commonTags() {
        List<Tag> tags = new ArrayList<>();
        tags.add(Tag.of("application", myProperties.getApplicationName()));
        tags.add(Tag.of("node", myProperties.getNodeName()));
        return tags;
    }
}

The problem is that my metrics behave correctly and even some of the Boot's metrics (at least http.server.requests ) see my tags. But jvm.* , system.* , tomcat.* and many others still don't have the needed tags.

Answer 1

If you are looking for common tags support, you can do it by registering a MeterFilter doing it.

See this commit or this branch for an example.

With the upcoming Spring Boot 2.1.0.M1 , you can use the following properties:

management.metrics.tags.*= # Common tags that are applied to every meter.

See the reference for details.

UPDATED:

As the question has been updated, I checked the updated question with this MeterFilter -based approach and confirmed it's working as follows:

Request: http://localhost:8080/actuator/metrics/jvm.gc.memory.allocated

Response:

 { "name" : "jvm.gc.memory.allocated", "measurements" : [ { "statistic" : "COUNT", "value" : 1.98180864E8 } ], "availableTags" : [ { "tag" : "stack", "values" : [ "prod" ] }, { "tag" : "region", "values" : [ "us-east-1" ] } ] }

I didn't check the approach which has been provided in the updated question but I'd just use the proven MeterFilter -based approach unless there's any reason to stick with the approach.

2nd UPDATED:

I looked into the approach and was able to reproduce it with this branch .

It's too late to apply common tags in @PostConstruct as some metrics have been registered already. The reason why http.server.requests works is that it will be registered with the first request. Try to put a breakpoint on the point of filters' application if you're interested in it.

In short, try the above approach which is similar to the upcoming Spring Boot out-of-box support.

Answer 2

For Server:
(ie @RestController )
To add custom metrics tags for reactive-spring-boot application, in addition to default tags provided by the spring-boot framework.
provide one or more @Beans that implement WebFluxTagsContributor
OR
To replace the default tags, provide a @Bean that implements WebFluxTagsProvider .
Reference implementation which uses Spring-WebMVC as one of the answers.
Ref:

Following implementation is in groovy.

import io.micrometer.core.instrument.Tag
import org.springframework.boot.actuate.metrics.web.reactive.server.WebFluxTagsContributor
import org.springframework.http.HttpMethod
import org.springframework.http.HttpStatus
import org.springframework.stereotype.Component
import org.springframework.web.server.ServerWebExchange

@Component
class CustomWebClientExchangeTagsProvider implements WebFluxTagsContributor {

    final KEY = "key"

    /**
     * Provides tags to be associated with metrics for the given {@code exchange}.
     * @param exchange the exchange
     * @param ex the current exception (maybe {@code null})
     * @return tags to associate with metrics for the request and response exchange
     */
    @Override
    Iterable<Tag> httpRequestTags(ServerWebExchange exchange, Throwable ex) {
        String apiKey = exchange.request.queryParams[KEY] ?: "default_api_key"
        HttpStatus status = exchange.response.statusCode ?: HttpStatus.INTERNAL_SERVER_ERROR
        HttpMethod method = exchange.request.method ?: HttpMethod.OPTIONS
        String group = (status.value() >= 500 ? "5XX" : (status.value() >= 400) ? "4XX" : (status.value() >= 300) ? "3XX" : "NONE")

        Tag statusTag = Tag.of("status", status.value().toString())
        Tag methodTag = Tag.of("method", method.toString())
        Tag apiKeyTag = Tag.of(KEY, apiKey)
        Tag groupTag = Tag.of("group", group)

        return Arrays.asList(statusTag, methodTag, apiKeyTag, groupTag)
    }
}

Metrics tags:

Verify for client: metric http.client.requests.percentile or http.client.requests .

http://localhost:8010/metrics/http.client.requests

Verify for server:

http://localhost:8010/metrics/http.server.requests

similarly, tags will be added for http.server.requests.percentile metrics

{
    "name": "http.server.requests",
    "description": null,
    "base_unit": "milliseconds",
    "measurements": [
        {
            "statistic": "COUNT",
            "value": 17.0
        },
        {
            "statistic": "TOTAL_TIME",
            "value": 8832.186054
        },
        {
            "statistic": "MAX",
            "value": 6.514132
        }
    ],
    "available_tags": [
        {
            "tag": "exception",
            "values": [
                "None",
                "ResponseStatusException"
            ]
        },
        {
            "tag": "method",
            "values": [
                "GET"
            ]
        },
        {
            "tag": "application",
            "values": [
                "myapplication"
            ]
        },
        {
            "tag": "uri",
            "values": [
                "/myapplication/v1/abcd",
                "/manage/metrics/{requiredMetricName}",
                "/manage",
                "/manage/metrics",
                "/myapplication/v1/windows",
                "/**"
            ]
        },
        {
            "tag": "outcome",
            "values": [
                "CLIENT_ERROR",
                "SERVER_ERROR",
                "SUCCESS"
            ]
        },
        {
            "tag": "key",
            "values": [
                "default_api_key",
                "[abcd]"
            ]
        },
        {
            "tag": "status",
            "values": [
                "404",
                "200",
                "502"
            ]
        },
        {
            "tag": "group",
            "values": [
                "4XX",
                "NONE",
                "5XX"
            ]
        }
    ]
}

OR

If you are using prometheus
you can verify custom tags as follows

http://localhost:8010/prometheus

Client:

http_client_requests_seconds_count{application="myapplication",clientName="myhostname.com",method="POST",outcome="CLIENT_ERROR",status="403",uri="/urlIamTryingToHit/v1/list",} 1.0

Not added for a client so there is no custom tags only existing tags provided by spring-boot.

Server:

http_server_requests_seconds_sum{application="myapplication",exception="None",group="4XX",key="default_api_key",method="GET",outcome="CLIENT_ERROR",status="404",uri="/manage/metrics/{requiredMetricName}",} 0.004154207

We can observe group="4XX" , key="default_api_key" , method="GET" , status="404" along with existing default tags.

---

For Client:
To customize the tags, and depending on your choice of client, you can provide a @Bean that implements RestTemplateExchangeTagsProvider or WebClientExchangeTagsProvider .
There are convenience static functions in RestTemplateExchangeTags and WebClientExchangeTags .
Ref:

Answer 3

If you use spring-boot-actuator and you need to add dynamical tags to default metrics simply define a WebMvcTagsProvider bean:
Default implementation of WebMvcTagsProvider.
@author Jon Schneider
@since 2.0.0

public class DefaultWebMvcTagsProvider implements WebMvcTagsProvider {

@Override
public Iterable<Tag> getTags(HttpServletRequest request, HttpServletResponse response, Object handler,
        Throwable exception) {
    return Tags.of(WebMvcTags.method(request), WebMvcTags.uri(request, response), WebMvcTags.exception(exception),
            WebMvcTags.status(response), WebMvcTags.outcome(response));
}

@Override
public Iterable<Tag> getLongRequestTags(HttpServletRequest request, Object handler) {
    return Tags.of(WebMvcTags.method(request), WebMvcTags.uri(request, null));
}

}

Answer 4

In response to comment under my original question, here is a small example of adding custom metrics:

package io.github.stepio.examples.metrics;

import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Tag;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;

@Component
public class MyImportantComponent {

    protected static final String CONNECTIONS_CURRENT = "connections.created";
    protected static final String CONNECTIONS_IDLE = "connections.idle";
    protected static final String CONNECTIONS_MAX = "connections.max";

    protected static final String PREFIX_REQUESTS_FOR_SERVICE = "requests";

    protected static final String SERVICE = "service";

    protected AtomicInteger connectionMax = new AtomicInteger();

    @Autowired
    protected MeterRegistry meterRegistry;

    @Override
    public List<Tag> tags() {
        return new ArrayList<>();
    }

    @Override
    public void trackConnectorsCurrent(AtomicInteger counter) {
        this.meterRegistry.gauge(CONNECTIONS_CURRENT, tags(), counter);
    }

    @Override
    public void trackConnectorsIdle(Collection<?> collection) {
        this.meterRegistry.gaugeCollectionSize(CONNECTIONS_IDLE, tags(), collection);
    }

    @Override
    public void submitConnectorsMax(final int value) {
        this.meterRegistry.gauge(CONNECTIONS_MAX, tags(), this.connectionMax).set(value);
    }

    @Override
    public void incrementRequestsForService(String serviceName) {
        this.meterRegistry.counter(PREFIX_REQUESTS_FOR_SERVICE, tags(SERVICE, serviceName)).increment();
    }

}