How to query container memory limit in Prometheus

Question

I am using Prometheus tool for monitoring my Kube.netes cluster.

I have set a resource limit(memory limit) in my deployments and need to configure a panel for showing the total memory available. Please let me know the query needed to run in Prometheus for getting the total memory limit available for my deployment.

Answer 1

It is possible using metrics kube_pod_container_resource_limits_memory_bytes (provided by kube-state-metrics ) and container_memory_usage_bytes (provided by kubelet/cAdvisor)

label_replace(
  label_replace(
    kube_pod_container_resource_limits_memory_bytes{},
    "pod_name", 
    "$1", 
    "pod", 
    "(.+)"
  ),
  "container_name", 
  "$1", 
  "container", 
  "(.+)"
) 
-
on(pod_name,namespace,container_name) 
avg(
      container_memory_usage_bytes{pod_name=~".+"}
)
by (pod_name,namespace,container_name)

A little explanation of the query: It is a subtraction of the memory limit and the actual usage. label_replace functions are needed to match the label names of both metrics, as they are obtained from different targets. avg is used to get the average between pod restarts, as every pod restart creates a new metric. {pod_name=~".+"} is used to filter metrics from container_memory_usage_bytes that are not useful for this case

Answer 2

The following PromQL query returns per-container free memory starting from Kube.netes v1.16:

kube_pod_container_resource_limits{resource="memory"}
  - on(namespace, pod, container)
container_memory_usage_bytes

The on() modifier instructs Prometheus to take into account only the listed labels when finding pairs of time series with identical labels on the left and the right side of - operator. See these docs for details.