Prometheus - Adding new labels to gauge results in an `inconsistent label cardinality` error

Question

I have a Go app that sends data to a prometheus gauge

...
import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)
...
var gauge = promauto.NewGaugeVec(prometheus.GaugeOpts{
    Name: "some_name",
    Help: "some desc",
},
    []string{"labelA", "labelB"},
)
...
// sending data to gauge
gauge.With(prometheus.Labels{
  "labelA": "...",
  "labelB": "...",
})

I then modified the app to include a third label ( labelC )

...
var gauge = promauto.NewGaugeVec(prometheus.GaugeOpts{
    Name: "some_name",
    Help: "some desc",
},
    []string{"labelA", "labelB", "labelC"},
)
...
gauge.With(prometheus.Labels{
  "labelA": "...",
  "labelB": "...",
  "labelC": "...",
})

But now when I run the app that contains the new label, I get this error

panic: inconsistent label cardinality: expected ... label values but got ... in prometheus.Labels{...}

the error happens when calling gauge.With

Anyone has any idea why?

Answer 1

The client library will throw this error if the number of labels in With doesn't match the number of labels in NewGaugeVec . So you likely forgot to add labelC: "..." somewhere in your code. You should be able to find the line in the stack trace.