Does the rate function really give average over time in prometheus?

Question

Does rate function really give average over time?

I send 20 requests into an endpoint by

ab -n 20 http://0.0.0.0:8001/

Snapshot

So, I use rate function with a metrics over 20s, so it should give me 1 value because there are 20 requests over the last 20s.

So 20 / 20 = 1 but it provides the value 2 .

I believe there has no relation between scrape_interval and evaluation_interval for the result, my both interval is 10s

Answer 1

Usually rate(m[d]) returns the average per-second change rate for the counter m over the previous time interval d . But sometimes Prometheus can return unexpected results from rate() function because of extrapolation. See this issue for details. Some Prometheus-compatible query engines such as MetricsQL try solving this issue - see this comment and this article for technical details.

Prometheus is going to solve this issue too - see this design doc .

Answer 2

If your scrape interval is 10s, then this is expected. The way this works is that Prometheus takes the 2 samples in your 20s interval (as they're 10s apart), calculates the difference (20), extrapolates that to the whole interval (40), then divides by the length of the interval (20) so you get 2.

I don't like this either and I've been advocating for a better rate implementation, which looks at the last sample before the range and the last sample in the range (so you would have an increase of 20 over 20s in your case, rather than an increase of 20 over 10s or possibly an increase of 0 over 10s, depending when you happen to query). But so far that has gone nowhere. So for now at least, welcome to the club.

One exceedingly hacky way to work against Prometheus' implementation is to reverse engineer it. Eg the expression that would give you the actual rate over 20 seconds in your case is:

rate(hello_worlds_total[30s]) / 30 * 20

Ie Prometheus takes the rate over 20 seconds, extrapolates it to 30, then you undo that extrapolation. But it requires you to be aware of the scrape interval and do the math to undo Prometheus' extrapolation.