K8S Pod OOM 因明显的 memory 泄漏而被杀死，memory go 在哪里？

Question

我遇到 K8S POD 被 OOM 杀死的问题，但有一些奇怪的情况和观察结果。

pod 是一个基于 golang 1.15.6 的 REST 服务，运行在 X86 64 位架构上。 当 pod 在基于 VM 的集群上运行时，一切正常，服务运行正常。 当服务在直接在硬件上配置的节点上运行时，它似乎会遇到 memory 泄漏并最终出现 OOMed。

观察发现，当在有问题的配置上运行时，“kubectl top pod”将报告 memory 利用率不断增加，直到达到定义的限制 (64MiB)，此时 OOM killer 被调用。

使用“top”从 pod 内部进行的观察表明 pod 内部各种进程的 memory 使用情况稳定，使用大约 40MiB RSS。 top 报告的 VIRT、RES、SHR 的值随着时间的推移保持稳定，只有很小的波动。

我广泛分析了 golang 代码，包括随着时间的推移获得 memory 个配置文件 (pprof)。 实际的 golang 代码没有泄漏的迹象，这与基于 VM 的环境中的正确操作和来自顶部的观察结果相符。

下面的 OOM 消息还表明 pod 使用的 RSS 总量仅为 38.75MiB（总和/RSS = 9919 页 *4k = 38.75MiB）。

kernel: [651076.945552] xxxxxxxxxxxx invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=999
kernel: [651076.945556] CPU: 35 PID: 158127 Comm: xxxxxxxxxxxx Not tainted 5.4.0-73-generic #82~18.04.1
kernel: [651076.945558] Call Trace:
kernel: [651076.945567]  dump_stack+0x6d/0x8b
kernel: [651076.945573]  dump_header+0x4f/0x200
kernel: [651076.945575]  oom_kill_process+0xe6/0x120
kernel: [651076.945577]  out_of_memory+0x109/0x510
kernel: [651076.945582]  mem_cgroup_out_of_memory+0xbb/0xd0
kernel: [651076.945584]  try_charge+0x79a/0x7d0
kernel: [651076.945585]  mem_cgroup_try_charge+0x75/0x190
kernel: [651076.945587]  __add_to_page_cache_locked+0x1e1/0x340
kernel: [651076.945592]  ? scan_shadow_nodes+0x30/0x30
kernel: [651076.945594]  add_to_page_cache_lru+0x4f/0xd0
kernel: [651076.945595]  pagecache_get_page+0xea/0x2c0
kernel: [651076.945596]  filemap_fault+0x685/0xb80
kernel: [651076.945600]  ? __switch_to_asm+0x40/0x70
kernel: [651076.945601]  ? __switch_to_asm+0x34/0x70
kernel: [651076.945602]  ? __switch_to_asm+0x40/0x70
kernel: [651076.945603]  ? __switch_to_asm+0x34/0x70
kernel: [651076.945604]  ? __switch_to_asm+0x40/0x70
kernel: [651076.945605]  ? __switch_to_asm+0x34/0x70
kernel: [651076.945606]  ? __switch_to_asm+0x40/0x70
kernel: [651076.945608]  ? filemap_map_pages+0x181/0x3b0
kernel: [651076.945611]  ext4_filemap_fault+0x31/0x50
kernel: [651076.945614]  __do_fault+0x57/0x110
kernel: [651076.945615]  __handle_mm_fault+0xdde/0x1270
kernel: [651076.945617]  handle_mm_fault+0xcb/0x210
kernel: [651076.945621]  __do_page_fault+0x2a1/0x4d0
kernel: [651076.945625]  ? __audit_syscall_exit+0x1e8/0x2a0
kernel: [651076.945627]  do_page_fault+0x2c/0xe0 
kernel: [651076.945628]  page_fault+0x34/0x40
kernel: [651076.945630] RIP: 0033:0x5606e773349b 
kernel: [651076.945634] Code: Bad RIP value.
kernel: [651076.945635] RSP: 002b:00007fbdf9088df0 EFLAGS: 00010206
kernel: [651076.945637] RAX: 0000000000000000 RBX: 0000000000004e20 RCX: 00005606e775ce7d
kernel: [651076.945637] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00007fbdf9088dd0
kernel: [651076.945638] RBP: 00007fbdf9088e48 R08: 0000000000006c50 R09: 00007fbdf9088dc0
kernel: [651076.945638] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fbdf9088dd0
kernel: [651076.945639] R13: 0000000000000000 R14: 00005606e7c6140c R15: 0000000000000000
kernel: [651076.945640] memory: usage 65536kB, limit 65536kB, failcnt 26279526
kernel: [651076.945641] memory+swap: usage 65536kB, limit 9007199254740988kB, failcnt 0
kernel: [651076.945642] kmem: usage 37468kB, limit 9007199254740988kB, failcnt 0
kernel: [651076.945642] Memory cgroup stats for /kubepods/burstable/pod34ffde14-8e80-4b3a-99ac-910137a04dfe:
kernel: [651076.945652] anon 25112576
kernel: [651076.945652] file 0
kernel: [651076.945652] kernel_stack 221184
kernel: [651076.945652] slab 41406464
kernel: [651076.945652] sock 0
kernel: [651076.945652] shmem 0
kernel: [651076.945652] file_mapped 2838528
kernel: [651076.945652] file_dirty 0
kernel: [651076.945652] file_writeback 0 
kernel: [651076.945652] anon_thp 0
kernel: [651076.945652] inactive_anon 0
kernel: [651076.945652] active_anon 25411584
kernel: [651076.945652] inactive_file 0
kernel: [651076.945652] active_file 536576
kernel: [651076.945652] unevictable 0
kernel: [651076.945652] slab_reclaimable 16769024
kernel: [651076.945652] slab_unreclaimable 24637440
kernel: [651076.945652] pgfault 7211542
kernel: [651076.945652] pgmajfault 2895749
kernel: [651076.945652] workingset_refault 71200645
kernel: [651076.945652] workingset_activate 5871824
kernel: [651076.945652] workingset_nodereclaim 330
kernel: [651076.945652] pgrefill 39987763
kernel: [651076.945652] pgscan 144468270 
kernel: [651076.945652] pgsteal 71255273 
kernel: [651076.945652] pgactivate 27649178
kernel: [651076.945652] pgdeactivate 33525031
kernel: [651076.945653] Tasks state (memory values in pages):
kernel: [651076.945653] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name   
kernel: [651076.945656] [ 151091]     0 151091      255        1    36864        0          -998 pause  
kernel: [651076.945675] [ 157986]     0 157986       58        4    32768        0           999 dumb-init  
kernel: [651076.945676] [ 158060]     0 158060    13792      869   151552        0           999 su  
kernel: [651076.945678] [ 158061]  1234 158061    18476     6452   192512        0           999 yyyyyy
kernel: [651076.945679] [ 158124]  1234 158124     1161      224    53248        0           999 sh  
kernel: [651076.945681] [ 158125]  1234 158125   348755     2369   233472        0           999 xxxxxxxxxxxx
kernel: [651076.945682] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=a0027a4fe415aa7a6ad54aa3fbf553b9af27c61043d08101931e985efeee0ed7,mems_allowed=0-3,oom_memcg=/kubepods/burstable/pod34ffde14-8e80-4b3a-99ac-910137a04dfe,task_memcg=/kubepods/burstable/pod34ffde14-8e80-4b3a-99ac-910137a04dfe/a0027a4fe415aa7a6ad54aa3fbf553b9af27c61043d08101931e985efeee0ed7,task=yyyyyy,pid=158061,uid=1234
kernel: [651076.945695] Memory cgroup out of memory: Killed process 158061 (yyyyyy) total-vm:73904kB, anon-rss:17008kB, file-rss:8800kB, shmem-rss:0kB, UID:1234 pgtables:188kB oom_score_adj:999
kernel: [651076.947429] oom_reaper: reaped process 158061 (yyyyyy), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

OOM 消息明确指出 usage = 65536kB，limit = 65536kB，但我并没有立即了解 memory 中未在 RSS 下计算的大约 25MiB 已经消失的地方。

我看到 slab_unreclaimable = 24637440, (24MiB)，这大约是似乎下落不明的 memory 的数量，但不确定这是否有任何重要意义。

寻找有关在哪里使用 memory 的任何建议。 欢迎任何意见。

Answer 1

I see slab_unreclaimable = 24637440, (24MiB), which is approximately the amount of memory that appears to be unaccounted for...

有关平板的详细信息，您可以尝试命令slabinfo或执行cat /proc/slabinfo 。 该表可以向您指出 memory 去了哪里。