Prometheus 长周期费率查询

Question

When using the rate function over a long period (eg 7d ) I am getting the error "query processing would load too many samples into memory in query execution" .

My query is

histogram_quantile(0.90, rate(http_request_in_seconds_bucket[7d]))

Answer 1

This error happens because Prometheus has a limit of samples it can process in memory.

I solved this with subqueries which was added on Prometheus 2.7 . This allows you to separately query smaller time intervals and then aggregate them together.

For example, I changed my query to multiple 24 hours subqueries and then averaged it together

histogram_quantile(0.90, avg_over_time(rate(http_request_in_seconds_bucket[24h])[7d:12h]))

There are other solutions:

1. Increase the number of samples it can process by changing the flag query.max-samples (not recommended)
2. Use Recording Rules - recommended over subqueries as it reduces the load on prometheus servers, but takes more efforts to set up.