How to get cpu and memory usage of nodes/pods in prometheus?

Question

As beginener, I have tried k9s and kubernetes ' kubectl top nodes ',for the cpu and memory usage and values are matched.Meanwhile I tried with prometheus UI, with ' avg(container_cpu_user_seconds_total{node="dev-node01"}) ' and ' avg(container_cpu_usage_seconds_total{node="dev-node01"}) ' for dev-node01. I cant get matching values.Any help will be appreciated as I am beginner.please any help would be appreciated.

Answer 1

if metrics 'container_cpu_user_seconds_total' showing output then it should work. I used the same query which you mentioned above and it's working for me. Check the graph and console tab as well in Prometheus.

Please try this

avg(container_cpu_user_seconds_total{node="NODE_NAME"})

Answer 2

container_cpu_usage_seconds_total is a counter , so you'll want to read up on what that means and how to query a counter.

In this case, you'll likely want to use the rate function, documented here .

rate(v range-vector) calculates the per-second average rate of increase of the time series in the range vector... rate should only be used with counters... Note that when combining rate() with an aggregation operator, always take a rate() first, then aggregate.

An example query for per-pod CPU usage would be:

sum(rate(container_cpu_usage_seconds_total{}[1h])) by (pod_name, namespace)

Also, you may want to check out Kubecost for k8s allocation metrics and APIs.

Answer 3

The following PromQL query returns CPU usage (as the number of used CPU cores) by pod in every namespace:

sum(rate(container_cpu_usage_seconds_total{container!=""})) by (namespace, pod)

The container!="" filter is needed for filtering out cgroups hierarchy metrics - see this answer for details.

The following query returns memory usage by pod in every namespace:

sum(container_memory_usage_bytes{container!=""}) by (namespace, pod)

The following query returns CPU usage (as the number of used CPU cores) by node:

sum(rate(container_cpu_usage_seconds_total{container!=""})) by (node)

The following query returns memory usage by node:

sum(container_memory_usage_bytes{container!=""}) by (node)