Adding labels to the default Golang Prometheus metrics

Question

I am currently using github.com/prometheus/client_golang to serve as an endpoint for retrieving metrics for my Golang appplication. It provides many default datasets out of the box, for example:

go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 10
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.13.10"} 1

It seems like I can not find any functionality in the library to add labels to these datasets. Because I'll have many of these applications running on the same machine, I need to add labels to differentiate the datapoints. Is there any way to do this in the client_golang library?

Answer 1

Each one of those applications should be scraped by prometheus as a separate job/instance. It will add the job label and an instance label which can also differentiate the different processes (and allow you to differentiate multiple instances of the same job).

See https://prometheus.io/docs/concepts/jobs_instances/#automatically-generated-labels-and-time-series for further details.

You could also use relabelling rules to add additional labels, depending on how you discover the applications. If you are using a static config ( https://prometheus.io/docs/prometheus/latest/configuration/configuration/#static_config ), you can add extra differentiating labels in that config. Similarly for file_sd_config : https://prometheus.io/docs/prometheus/latest/configuration/configuration/#file_sd_config

Answer 2

There's an option to add constant labels. EG:

var (
    labels = map[string]string{"application": "foobar"}

    // Status Metrics
    StateCalls = prometheus.NewCounter(prometheus.CounterOpts{
        Name:        "state_calls",
        Help:        "",
        ConstLabels: labels,
    })
)