Prometheus query to get memory limit commitment for the entire cluster

Question

I'm using the latest prometheus 2.21.0 and latest node-exporter

Trying to run the query and getting no datapoints found however both metrics kube_pod_container_resource_limits_memory_bytes and node_memory_MemTotal_bytes are working independently and return data

(sum(kube_pod_container_resource_limits_memory_bytes) / :node_memory_MemTotal_bytes:sum)*100

So two questions

1. I never saw such syntax before :node_memory_MemTotal_bytes:sum - is it valid prometheus query?
2. What is wrong with the query if the syntax is correct?

Answer 1

1. This is a convention widely used in prometheus land. It means this metric is not one directly scraped from some target(s), but instead a result of recording rule. This convention is described here .
2. If queries on both left and right side return data individually but after performing artihmetic on them you are left with no data then it probably means labels on them are not exactly the same. Execute them separately and compare labels you have on your results. Assuming that :node_memory_MemTotal_bytes:sum does return data then you'll probably have to add sum there too to remove any remaining labels there