cgroup rss的memory远高于docker容器中所有进程的memory使用率汇总

Question

I hava a Redis runing in a container. Inside the container cgroup rss show using about 1283MB memory.

The kmem memory usage is 30.75MB.

The summary of the memory usage of all processes in the docker container is 883MB.

How can i figure out the "disappeared memory "(1296-883-30=383MB).The "disappeared memory" will growing with the time pass.Flinally the container will be oom killed.

environmet info is

redis version:4.0.1

docker version:18.09.9

k8s version:1.13

**the memory usage is 1283MB **

root@redis-m-s-rbac-0:/opt#cat /sys/fs/cgroup/memory/memory.usage_in_bytes 
1346289664  >>>> 1283.921875 MB

the kmem memory usage is 30.75MB

root@redis-m-s-rbac-0:/opt#cat /sys/fs/cgroup/memory/memory.kmem.usage_in_bytes 
32194560  >>> 30.703125 MB

root@redis-m-s-rbac-0:/opt#cat /sys/fs/cgroup/memory/memory.stat 
cache 3358720
rss 1359073280 >>>  1296.11328125 MB
rss_huge 515899392
shmem 0
mapped_file 405504
dirty 0
writeback 0
swap 0
pgpgin 11355630
pgpgout 11148885
pgfault 25710366
pgmajfault 0
inactive_anon 0
active_anon 1359245312
inactive_file 2351104
active_file 1966080
unevictable 0
hierarchical_memory_limit 4294967296
hierarchical_memsw_limit 4294967296
total_cache 3358720
total_rss 1359073280
total_rss_huge 515899392
total_shmem 0
total_mapped_file 405504
total_dirty 0
total_writeback 0
total_swap 0
total_pgpgin 11355630
total_pgpgout 11148885
total_pgfault 25710366
total_pgmajfault 0
total_inactive_anon 0
total_active_anon 1359245312
total_inactive_file 2351104
total_active_file 1966080
total_unevictable 0

**the summary of the memory usage of all processes in the docker container is 883MB **

root@redis-m-s-rbac-0:/opt#ps aux | awk '{sum+=$6} END {print sum / 1024}'
883.609

Answer 1

This is happening because usage_in_bytes does not show exact value of memory and swap usage. The memory.usage_in_bytes show current memory(RSS+Cache) usage.

5.5 usage_in_bytes For efficiency, as other kernel components, memory cgroup uses some optimization to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the method and doesn't show 'exact' value of memory (and swap) usage, it's a fuzz value for efficient access. (Of course, when necessary, it's synchronized.) If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP) value in memory.stat(see 5.2).

Reference: https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt