clear prometheus metrics from collector

Question

I'm trying to modify prometheus mesos exporter to expose framework states: https://github.com/mesos/mesos_exporter/pull/97/files

A bit about mesos exporter - it collects data from both mesos /metrics/snapshot endpoint, and /state endpoint. The issue with the latter, both with the changes in my PR and with existing metrics reported on slaves, is that metrics created lasts for ever (until exporter is restarted). So if for example a framework was completed, the metrics reported for this framework will be stale (eg it will still show the framework is using CPU).

So I'm trying to figure out how I can clear those stale metrics. If I could just clear the entire mesosStateCollector each time before collect is done it would be awesome. There is a delete method for the different p8s vectors (eg GaugeVec ), but in order to delete a metric, I need to not only the label name, but also the label value for the relevant metric.

Answer 1

Ok, so seems it was easier than I thought (if only I was familiar with go-lang before approaching this task). Just need to cast the collector to GaugeVec and reset it:

        prometheus.NewGaugeVec(prometheus.GaugeOpts{
            Help:      "Total slave CPUs (fractional)",
            Namespace: "mesos",
            Subsystem: "slave",
            Name:      "cpus",
        }, labels): func(st *state, c prometheus.Collector) {
            c.(*prometheus.GaugeVec).Reset() ## <-- added  this for each GaugeVec
            for _, s := range st.Slaves {
                c.(*prometheus.GaugeVec).WithLabelValues(s.PID).Set(s.Total.CPUs)
            }
        },