redis 內存使用超過 k8s 資源限制

Question

k8s集群中部署了一個redis實例，內存限制為1G：

resources:
  limits:
    memory: 1Gi

並且還設置了redis maxmemory為1G，實際內存使用量為~550M：

info memory
# Memory
used_memory:577791152
used_memory_human:551.02M
used_memory_rss:611942400
used_memory_rss_human:583.59M
used_memory_peak:578877888
used_memory_peak_human:552.06M
used_memory_peak_perc:99.81%
used_memory_overhead:32108388
used_memory_startup:786608
used_memory_dataset:545682764
used_memory_dataset_perc:94.57%
total_system_memory:67555897344
total_system_memory_human:62.92G
used_memory_lua:41984
used_memory_lua_human:41.00K
maxmemory:1000000000
maxmemory_human:953.67M
maxmemory_policy:noeviction
mem_fragmentation_ratio:1.06
mem_allocator:jemalloc-4.0.3
active_defrag_running:0
lazyfree_pending_objects:0

我觀察到 redis 容器內存使用量超過了 prometheus 的 1G (metric sum by(container_name) (container_memory_working_set_bytes{namespace="$namespace",container_name!="",container_name=~"$container",container_name!="POD",pod_name="$pod"}) ）：

那么問題來了：

1. 如何釋放內存？
2. container_memory_working_set_bytes和 redis 報告used_memory_rss used_memory什么關系？

Answer 1

這是一個與 k8s prometheus issue相關的問題，prometheus 實際上得到了翻倍的結果：

container_cpu_usage_seconds_total{container="redis-server",container_name="redis-server",cpu="total",endpoint="https-metrics",id="/kubepods/pod149c0943-bc55-4e5d-ac9d-59fedca56138/1e2f5a63ddba49cb13465383e0de66e50ce9f084b52975f9bea8d1d77b966b3c",image="sha256:8a74675d2c029efb8c574f6097f2494d0e6acb7427aeea874227be48d96a3fb2",instance="192.168.116.119:10250",job="expose-kubelets-metrics",name="k8s_redis-server_redis-server-0_user-data-service-test_149c0943-bc55-4e5d-ac9d-59fedca56138_0",namespace="user-data-service-test",node="k8sworker03",pod="redis-server-0",pod_name="redis-server-0",service="expose-kubelets-metrics"}

container_cpu_usage_seconds_total{container="redis-server",container_name="redis-server",endpoint="https-metrics",instance="192.168.116.119:10250",job="expose-kubelets-metrics",namespace="user-data-service-test",node="k8sworker03",pod="redis-server-0",pod_name="redis-server-0",service="expose-kubelets-metrics"}    

所以結果是雙倍的。