Average Memory Usage Query - Prometheus

Question

How do I write a query that outputs average memory usage for instances over the past 24 hours? The following query displays the current memory usage

100 * (1 - ((node_memory_MemFree + node_memory_Cached + node_memory_Buffers) / node_memory_MemTotal))

For CPU, I was able to use irate

100 * (1 - avg(irate(node_cpu[24h])) BY (instance))

How do I use irate and avg for memory?

Answer 1

average memory usage for instances over the past 24 hours

You can use avg_over_time :

100 * (1 - ((avg_over_time(node_memory_MemFree[24h]) + avg_over_time(node_memory_Cached[24h]) + avg_over_time(node_memory_Buffers[24h])) / avg_over_time(node_memory_MemTotal[24h])))

For CPU, I was able to use irate

irate only looks at the last two samples, and that query is the inverse of how many modes you have and will be constant (it's always 0.1 on my kernel). You want

100 - (avg by (instance) (rate(node_cpu{job="node",mode="idle"}[5m])) * 100)

Note that this is a 5 minute moving average and you can change [5m] to whatever period of time you are looking for such as [24h] .

Answer 2

Update commands

CPU Usage :

(1 - avg(irate(node_cpu_seconds_total{mode="idle"}[10m])) by (instance)) * 100

Memory Usage :

100 * (1 - ((avg_over_time(node_memory_MemFree_bytes[10m]) + avg_over_time(node_memory_Cached_bytes[10m]) + avg_over_time(node_memory_Buffers_bytes[10m])) / avg_over_time(node_memory_MemTotal_bytes[10m])))

Answer 3

Also, we can query with accounts;

CPU Usage for Windows

#Average
100 - (avg by (instance) (irate(windows_cpu_time_total{job=~"dev",mode="idle"}[2m])) * 100)
#Minimum
100 - (max by (instance) (irate(windows_cpu_time_total{job=~"dev",mode="idle"}[2m])) * 100)
#Maximum
100 - (min by (instance) (irate(windows_cpu_time_total{job=~"dev",mode="idle"}[2m])) * 100)

CPU Usage for Linux

#Average
100 - (avg by (instance) (irate(node_cpu_seconds_total{job="$job",mode="idle"}[5m])) * 100)
#Minimum
100 - (max by (instance) (irate(node_cpu_seconds_total{job="$job",mode="idle"}[5m])) * 100)
#Maximum
100 - (min by (instance) (irate(node_cpu_seconds_total{job="$job",mode="idle"}[5m])) * 100)

Memory Usage for Windows

#Average
100.0 - 100 * avg_over_time(windows_os_physical_memory_free_bytes{job=~"prod"}[30d]) / avg_over_time(windows_cs_physical_memory_bytes{job=~"prod"}[30d])
#Maximum
100.0 - 100 * min_over_time(windows_os_physical_memory_free_bytes{job=~"prod"}[30d]) / min_over_time(windows_cs_physical_memory_bytes{job=~"prod"}[30d])
#Minimum
100.0 - 100 * max_over_time(windows_os_physical_memory_free_bytes{job=~"prod"}[30d]) / max_over_time(windows_cs_physical_memory_bytes{job=~"prod"}[30d])

Memory Usage for Linux

#Avarege:
100 * (1 - ((avg_over_time(node_memory_MemFree_bytes{job=~"prod"}[30d]) + avg_over_time(node_memory_Cached_bytes{job=~"prod"}[30d]) + avg_over_time(node_memory_Buffers_bytes{job=~"prod"}[30d])) / avg_over_time(node_memory_MemTotal_bytes{job=~"prod"}[30d])))

#Maximum:
100 * (1 - ((min_over_time(node_memory_MemFree_bytes{job=~"prod"}[30d]) + min_over_time(node_memory_Cached_bytes{job=~"prod"}[30d]) + min_over_time(node_memory_Buffers_bytes{job=~"prod"}[30d])) / min_over_time(node_memory_MemTotal_bytes{job=~"prod"}[30d])))

#Minimum
100 * (1 - ((max_over_time(node_memory_MemFree_bytes{job=~"prod"}[30d]) + max_over_time(node_memory_Cached_bytes{job=~"prod"}[30d]) + max_over_time(node_memory_Buffers_bytes{job=~"prod"}[30d])) / max_over_time(node_memory_MemTotal_bytes{job=~"prod"}[30d])))

Note:- {job=~"prod"}, is account name. you can use your account names