Visualizing K8S CPU resources Prometheus and Grafana

Question

I'm new with monitoring tools like Prometheus and Grafana and I would like to create dashboard which represents current requests and limits resources and usage for a pod. In addition, this pod has 2 containers inside.

My resources for first container looks like:

 resources:
    requests:
      cpu: "3800m"
      memory: "9500Mi"
    limits:
      cpu: "6500m"
      memory: "9500Mi"

and for second container:

resources:
   limits:
     cpu: 100m
     memory: 100Mi
   requests:
     cpu: 50m
     memory: 50Mi

When executing this query in Prometheus:

rate(container_cpu_usage_seconds_total{pod=~"MY_POD"}[5m]) I get: And to be honest I dont know how this returned data are valid with resources. On Grafana it looks like this:

In addition I would like to add informations about requests and limits to dashboard, but I don't know how to scale dashboard to show all data.

When I execute this query: kube_pod_container_resource_requests{pod=~"MY_POD"} I get:

And this looks valid compared to my resources. I have proper value for limits too, but I would like to represent all this data (usage, requests, limits) on ona dashboard. Could somebody give my any tips how to achieve this?

Answer 1

Simple, just add 2 more queries

if you don't know where to add, below Metrics Browser and Options tab you can find + symbol to add more queries

and container_cpu_usage_seconds_total this metrics is a counter and this will give you the CPU Usage in Cores.

kube_pod_container_resource_requests{pod=~"MY_POD"} and kube_pod_container_resource_limits{pod=~"MY_POD"} . If it's only one Pod means no issues. But if you're having mutliple Pods means try to use Sum

• Query A: sum(rate(container_cpu_usage_seconds_total{pod=~"MY_POD"}[5m]))
• Query B: sum(kube_pod_container_resource_requests{pod=~"MY_POD"})
• Query C: sum(kube_pod_container_resource_limits{pod=~"MY_POD"})

This will looks good without too much details, for more details like container wise data just create three more Panel for Requests, Limits and Usage by Containers and add by(container) after the every Query

Another Approach:

Create Variables for Pod and Container so that you can select the Container which you want to see and add 3 queries in single panel so that the Panel looks more Dynamic and less Noise