如何区分两种不同的prometheus指标？

Question

Consider metric examples:

increase(application_executor_recordsWritten[20m])
increase(kafka_server_brokertopicmetrics_messagesin_total{topic="my_topic"}[20m])

If I execute those metrics separate on prometheus graph - everything works. But when try something like:

increase(application_executor_recordsWritten[20m]) -  increase(kafka_server_brokertopicmetrics_messagesin_total{topic="my_topic"}[20m])

I got No datapoints error .

1. May be it happens because application_executor_recordsWritten received for last 1 hour while kafka_server_brokertopicmetrics_messagesin_total is received for 6+ hours.

2. May it happens because those metric have different "gather settings", consider prometheus console output:

   application_executor_recordsWritten

   {app_name="app-name",exported_instance="application_111111111111111111",exported_job="application_111111111111111111",instance="XX.XXX.X.XX",job="job_name",number="1",role="executor"}

   kafka_server_brokertopicmetrics_messagesin_total

   {instance="XX.XXX.X.XX",job="job_name",topic="my_topic"}

Prometheus use something like ignore(???) keyword, but I can not figure out how does it work and how apply it for these metric.

Any ideas how to perform metrics difference? What is the correct syntax for this?

Answer 1

In PromQL, arithmetic binary operators between two metric ranges (aka vectors) are subject to vector matching : the operation is only applied on entries that have the same exact label set (name and avalue).

If there is aa difference and no values are paired, your get the infamous No data point error.

If you want to make them match, you have to

• either ignoring some labels that do not match ( metric1 - ignoring(a_lone_label) metric2 )
• or indicating on which label perform the match ( metric1 - on(common_label_name_and_value) metric2 )

In the examples you gave, it is unclear what should match. I would say instance and job ; it could be:

increase(application_executor_recordsWritten[...]) - on (job,instance) increase(kafka_server_brokertopicmetrics_messagesin_total{topic="my_topic"}[...])

If you have one side of the operator containing elements that should be paired with more than one element of the other side (call one-to-many matching), you have to indicate which side of the operator (right or left) has more entries: using group_<side:rigth|left> .