How to show aggregated CPU, RAM, Disk I/O usage of a cluster using Prometheus?

Question

I have 3 servers. I am using Prometheus with Node exporter and for visualizing Prometheus data I am using Grafana. The Node Exporter Full (1860) dashboard can only shows CPU, RAM, Disk I/O usages for each server individually. I want to get overall usages, for example, I want to get the whole cluster's (sarver1 + sarver2 + sarver3) aggregated CPU, RAM, and Disk I/O usages. What queries should I use to get the total cluster resources utilization?

Answer 1

Suppose you have the following query to get the CPU usage:

100 - (avg(irate(node_cpu_seconds_total{instance="server1",mode="idle"}[5m])) * 100)

To get the overall of several servers you need to change the query to the following:

100 - (avg(irate(node_cpu_seconds_total{instance=~"server1|server2|server3",mode="idle"}[5m])) * 100)