Golang Run GC on OOM

Question

Does golang run GC and retry allocation when it faces OOM?
We are facing an issue where the kubernetes pod (running the go code) gets OOM Killed when processing large files.

The go code processes many files, reading them line by line and for each line it parses (allocating some memory by create a parsed struct - local variable in the loop) and if matching some condition inserts it to a DB. This is run concurrently for multiple clients. This works fine with many clients having small files. But when it encounters a large sized file for a client it goes OOM.

We have the system running for a long time and don't see any memory leaks. Even the pprof analysis indicates no memory leak.

Answer 1

The OOM killer is the kernel's last resort when faced with memory pressure. "Either I sacrifice a process now, or the entire system may become unusable."

Because this is a last resort (for instance, the kernel will choose to reduce the amount of memory available for the filesystem cache long before invoking the OOM killer) the process that is chosen to be sacrificed has no say in this and there is no early warning ¹ .

In your case you are probably not running out of physical memory but hitting a user-defined memory limit for your container. But the kernel uses the same facilities to enforce this limit too.

You must raise your memory limit to account for the worst case workload, or improve your program's memory consumption to stay within its limit.

Given that small files work fine, it stands to reason that you're not limiting the concurrency appropriately. If you spawn one goroutine for every line all at once, in terms of memory usage there's no fundamental difference to reading the whole file. Concurrency can be easily limited with a semaphore

---

1. You can change the SIGKILL to a SIGTERM, but that won't help you here. Your ops team will hate you if you change the meaning of SIGTERM from "exit" to "release some memory to the OS but keep running". Not that Go has any way of doing so anyway.