普罗米修斯或使用rate（）时

Question

Summary

I'm trying to figure out how to properly use the OR | operator in a Prometheus query because my imported Grafana dashboard is not working.

Long version

I'm trying to debug a Grafana dashboard based on some data scraped from my Kubernetes pods running AppMetrics/Prometheus ; the dashboard is here . Basically what happens is that when the value "All" for the server is selected on the Grafana dashboard ( server is an individual pod in this case), no data appears. However, when I select an individual pod, then data does appear.

Here's an example of the same metric scraped from the two pods:

# HELP application_httprequests_transactions 
# TYPE application_httprequests_transactions summary
application_httprequests_transactions_sum{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test"} 5.006965628
application_httprequests_transactions_count{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test"} 1367
application_httprequests_transactions{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test",quantile="0.5"} 0.000202825
application_httprequests_transactions{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test",quantile="0.75"} 0.000279318
application_httprequests_transactions{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test",quantile="0.95"} 0.000329862
application_httprequests_transactions{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test",quantile="0.99"} 0.055584233

# HELP application_httprequests_transactions 
# TYPE application_httprequests_transactions summary
application_httprequests_transactions_sum{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test"} 6.10214788
application_httprequests_transactions_count{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test"} 1363
application_httprequests_transactions{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test",quantile="0.5"} 0.000218548
application_httprequests_transactions{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test",quantile="0.75"} 0.000277483
application_httprequests_transactions{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test",quantile="0.95"} 0.033821094
application_httprequests_transactions{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test",quantile="0.99"} 0.097113234

I ran the Query inspector in Grafana to find out which query it is calling, and then ran the PromQL query in Prometheus itself. Basically, when I execute the following PromQL queries individually, they return data:

rate(application_httprequests_transactions_count{env="test",app="MyApp",server="myapp-test-58d94bf78d-l9tdv"}[15m])*60

rate(application_httprequests_transactions_count{env="test",app="MyApp",server="myapp-test-58d94bf78d-jdq78"}[15m])*60

However, when I try to use PromQL's | operator to combine them, I don't get data back:

rate(application_httprequests_transactions_count{env="test",app="MyApp",server="myapp-test-58d94bf78d-l9tdv|myapp-test-58d94bf78d-jdq78"}[15m])*60

Here's the raw output from Grafana's query inspector:

xhrStatus:"complete"

request:Object

method:"GET"
url:"api/datasources/proxy/56/api/v1/query_range?query=rate(application_httprequests_transactions_count%7Benv%3D%22test%22%2Capp%3D%22MyApp%22%2Cserver%3D%22myapp-test-58d94bf78d-jdq78%7Cmyapp-test-58d94bf78d-l9tdv%7Cmyapp-test-5b8c9845fb-7lklm%7Cmyapp-test-5b8c9845fb-8jf7n%7Cmyapp-test-5b8c9845fb-d9x5c%7Cmyapp-test-5b8c9845fb-fw4gj%7Cmyapp-test-5b8c9845fb-vtl9z%7Cmyapp-test-5b8c9845fb-vv7xv%7Cmyapp-test-5b8c9845fb-wq9bs%7Cmyapp-test-5b8c9845fb-xqfrt%7Cmyapp-test-69999d58b5-549vd%7Cmyapp-test-69999d58b5-lmp8x%7Cmyapp-test-69999d58b5-nbvt9%7Cmyapp-test-69999d58b5-qphj2%7Cmyapp-test-6b8dcc5ffb-gjjvj%7Cmyapp-test-6b8dcc5ffb-rxfk2%7Cmyapp-test-7fdf446767-bzhm2%7Cmyapp-test-7fdf446767-hp46w%7Cmyapp-test-7fdf446767-rhqhq%7Cmyapp-test-7fdf446767-wxmm2%22%7D%5B1m%5D)*60&start=1540574190&end=1540574505&step=15"

response:Object

status:"success"

data:Object

resultType:"matrix"

result:Array[0] => []

I opened a GitHub issue for this as well; it has a quick GIF screen recording showing what I mean: AppMetrics/Prometheus#43

Answer 1

| is for regular expressions, PromQL doesn't have a | operator (but it does have an or operator). You need to specify that the matcher is a regex rather than an exact match with =~ :

rate(application_httprequest_transactions_count{env="test",app="MyApp",server=~"myapp-test-58d94bf78d-l9tdv|myapp-test-58d94bf78d-jdq78"}[15m])*60