Prometheus 查询以计算具有不同标签集的两个指标的百分比

Question

We have a service that accepts the HTTP request and responds with all the matching user data in the response and produces two metrics that are supposed to be made available as graphs/charts on the grafana. Metrics as mentioned here -

1. Total requests received by the application in different data centre regions

   application_requests_total{data_center_region_id=1} //200 application_requests_total{data_center_region_id=2} //100

2. Every request to the application will try and include all the matching user's data in the response metrics

   application_response_total{user="user1, data_center_region_id=1} //100 application_response_total{user="user1, data_center_region_id=2} //100 application_response_total{user="user2, data_center_region_id=1} //50 application_response_total{user="user2, data_center_region_id=2} //100

Quick observations on the metrics

• user label is only present in response metrics application_response_total
• data_center_region_id is the common label in request and response metrics
• One response can contain more than one user's data which is also reflected in the metrics application_response_total .

I need to find out the percentage of the responses at the user level against the total requests made to the application in specific data centre region.

Eg: Based on the above data the expected results would be

For data_center_region_id=1

• user1's data responded for 100/200 = 50% of the time
• user2's data responded for 50/200 = 25% of the time

For data_center_region_id=2

• user1's data responded for 100/100 = 100% of the time
• user2's data responded for 100/100 = 100% of the time

I tried a couple of queries based on the Prometheus vector matching documentation but couldn't achieve the expected results. Few samples queries as follows; I'm not sure but I think I messed up the ON / IGNORING and GROUP_LEFT / GROUP_RIGHT keyworkds

sum(rate(application_response_total{data_center_region_id=~"$region"}[5m])) by (user, data_center_region_id) / on(user) group_left(data_center_region_id) sum(rate(application_requests_total{data_center_region_id=~"$region"}[5m])) by (user, data_center_region_id)

Also took reference from the question here but nothing is working for me.

Please guide me with above-expected result;

Also, is this the only way to get the desired graphs?

Answer 1

Note that it may be a bad idea to put user into metric labels if the application is going to serve unlimited number of users. This may result in eg high cardinality issues .

As for the original question, you need to put common labels for the left-side and right-side series in the on(...) modifier, so Prometheus could find pairs of series with the given labels on both sides of / operator. The following query returns per-(user, region) rps share:

rate(application_response_total[5m])
  / on(data_center_region_id) group_left()
rate(application_requests_total[5m])

The group_left() modifier instructs Prometheus that the left side may contain multiple series with the identical data_center_region_id label value. In this case Prometheus independently divides each such series by the series at the right side with the matching data_center_rrgion_id label value. The resulting series will have the same set of labels as the left-side series. See these docs for details.