如何在 Prometheus PromQL 中划分 2 个指标

Question

I am building my dashboard in Graphana using Prometheus. I have 2 metrics (Total calls to a service and total timeout errors)

1 is total calls to a service PromQL

(increase(Fetching_RESPONSE_TIME_seconds_count{instance="${server}:8080"}[1h]) 

other is total timeout PromQL

(increase(dp_errors_total{code=~"12345",instance="${server}:8080"}[1h]))

I want to have one more column in my dashboard which shows percentage timeout which would be (total timeout*100/total calls to service).

when I do this PromQL

(increase(dp_errors_total{code=~"12345",instance="${server}:8080"}[1h])*100
/
(increase(Fetching_RESPONSE_TIME_seconds_count{instance="${server}:8080"}[1h])

It does not show anything to my dashboard.

How can I add one more column to my dashboard which would show percentage timeouts?

Answer 1

When you try to do an arithmetic expression Prometheus will try to match time series on the left and right side. It does it by labels they have. Both sides have to have the same labels (names and values). I don't know all the labels your time series have but I can guess that for example code label is only present on only dp_errors_total and not in the second one. I'd typically aggregate both operands first (by what is needed), for example:

sum by (server) ( ... dp_errors_total query ) 
/
sum by (server) ( ... Fetching_RESPONSE_TIME_seconds_count query ...)

or if there is only one server in $server then drop the by (server) part.

Answer 2

By default Prometheus performs the division for pairs of time series with identical sets of labels on the left and the right side of / operator. In our case time series on the left side of / contain code and instance labels, while time series on the right side of / contain only instance label. Prometheus cannot find matching pairs of time series, so it returns nothing according to these rules . This behavior can be changed with on() and group_left() modifiers:

• the on() modifier is used for limiting the set of labels, which are taken into account during searching for matching time series pairs
• the group_left() modifier is used for allowing matching multiple time series on the left side of / operator to a single time series on the right side. See these docs for more details.

So the resulting query should look like the following:

100 * increase(dp_errors_total{code=~"12345",instance="${server}:8080"}[1h])
  / on(instance) group_left()
increase(Fetching_RESPONSE_TIME_seconds_count{instance="${server}:8080"}[1h])