Prometheus - combining average used cpu query with nodes which have specific label

Question

I am trying to write query in Prometheus which gets the average used cpu of nodes with specific node label.

The used cpu query for a single host is:

(1-avg(irate(node_cpu_seconds_total{mode="idle",node=~host1"}[5m]))) * 100

The query which returns list of nodes with specific value is:

kube_node_labels{label_department="prod"}

How do I combine both queries?

Thanks in advance.

The output of both querires:

• node_cpu_seconds_total

{cpu="0",endpoint="metrics",host_ip="150.130.1.2",instance="150.130.1.2:2323",job="expose-node-metrics",mode="idle",namespace="prometheus",node="host001",pod="export-monitor",service="expose-node-metrics"}

• kube_node_labels

{endpoint="http",host_ip="150.130.1.2",instance="150.130.1.2:2323",job="expose-node-metrics",label_department="prod",label_beta_kubernetes_io_arch="amd64",label_beta_kubernetes_io_os="linux",label_beta_kubernetes_io_arch="amd64",label_kubernetes_io_hostname="host001",label_kubernetes_io_os="linux",label_node_role_kubernetes_io_worker="true",namespace="prometheus",node="host001",pod="export-monitor",service="expose-node-metrics"}

Answer 1

When you want to filter a metric from another metric, you can use binary set operators to select what you want. In this case, using AND :

vector1 and vector2 results in a vector consisting of the elements of vector1 for which there are elements in vector2 with exactly matching label sets.

You will notice that the node label is present in both metrics and that what you can use it to filter node metrics:

irate(node_cpu_seconds_total{mode="idle"}[5m]) AND ON(node) kube_node_labels{label_department="prod"}

Then, if you want to have the average by node:

avg(irate(node_cpu_seconds_total{mode="idle"}[5m]) AND ON(node) kube_node_labels{label_department="prod"}) BY(node)