Prometheus query to count unique label values over a period

Question

I want to calc the total distinct label counts over a period time.

Eg. I have below 4 sample data

Metrics Labels TimeStamp Values

cpu_usage{instance="192.168.100.10:20001",job="node2"}@1646225640 => 4
cpu_usage{instance="192.168.100.10:20001",job="node1"}@1646225700 => 5
cpu_usage{instance="192.168.100.10:20001",job="node3"}@1646225760 => 3
cpu_usage{instance="192.168.100.10:20001",job="node2"}@1646225820 => 4

So if I check startdate=1646225640, enddate=1646225700, I got 2 distinct jobs, which are node2 and node1

if I check startdate=1646225640, end date=1646225820, I got 3 distinct jobs, which are node1, node2, and node3.

Is there a way in promql can do this?

Answer 1

Find one way to achieve this

count by (job_temp) (count_over_time(cpu_usage[1h]))

or

sum(count by (job) (count_over_time(cpu_usage[1h])))

PromQL is a time-series based so I find it more useful to illustrate with image:

say we want to check between 1646225640 and 1646225820

prometheus data is based on metrics{labels} for a series of timestamp, so count_over_time will return results with 3 records:

instance=192.168.100.10:20001,job=node1 values=[[1646225700, 1]] instance=192.168.100.10:20001,job=node2 values=[[1646225640, 1], [1646225820, 1]] instance=192.168.100.10:20001,job=node3 values=[[1646225760, 1]]

for above results, if we count by job, it's like group by job, so we have 3 results too

if we count by labels that doesn't exist in the metrics, then prometheus will just take no label as group by filter, then we will only have 1 result cause they haven't been grouped at all, which is the second query sum method effect

Answer 2

The following PromQL query should return unique job label values for the metric cpu_usage on a time range (td... t] :

count(last_over_time(cpu_usage[d] @ t)) by (job)

This query uses the following PromQL features:

• count() aggregate function for counting the number of time series per each unique job value.
• last_over_time() rollup function, which returns the last value per each time series on the given lookbehind window d in square brackets.
• @ modifier for executing the query at the given timestamp t .

The query can be simplified if it is executed in recording rules or alerting rules . For example, the following query counts the number of unique time series per each unique job label value for the last hour:

count(last_over_time(cpu_usage[1h])) by (job)