Prometheus：减法标签匹配

Question

I have the following Prometheus query and want the right side of my substraction to only subtract the container_memory_usage_bytes for each node on the left side of the substraction.

Query:

sum(kube_node_status_capacity_memory_bytes) by (node) - on (instance) group_left container_memory_usage_bytes

The left side of the subtraction works.

But for the whole query I get

Error executing query: found duplicate series for the match group;many-to-many matching not allowed: matching labels must be unique on one side

How can I make my query to match exactly the labels node == instance from left to right side?

Answer 1

The RHS has no instance label, so it's trying to match all those series to one on the LHS . You have to also add sum before listing container_memory_usage_bytes metric, - sum(container_memory_usage_bytes) .

Please take a look here: prometheus-metrics .

Useful article: label-proportions .