如何在 Golang 中为进程设置内存限制

Question

I use syscall prlimit to set resource limit to process, it works for limit CPU time but when test memory usage, I come with the problem.

package sandbox

import (
    "syscall"
    "unsafe"
)

func prLimit(pid int, limit uintptr, rlimit *syscall.Rlimit) error {
    _, _, errno := syscall.RawSyscall6(syscall.SYS_PRLIMIT64, uintptr(pid), limit, uintptr(unsafe.Pointer(rlimit)), 0, 0, 0)
    var err error
    if errno != 0 {
        err = errno
        return err
    } else {
        return nil
    }
}

and it's my test.

func TestMemoryLimit(t *testing.T) {
    proc, err := os.StartProcess("test/memo", []string{"memo"}, &os.ProcAttr{})
    if err != nil {
        panic(err)
    }
    defer proc.Kill()
    var rlimit syscall.Rlimit
    rlimit.Cur = 10
    rlimit.Max = 10 + 1024
    prLimit(proc.Pid, syscall.RLIMIT_DATA, &rlimit)
    status, err := proc.Wait()
    if status.Success() {
        t.Fatal("memory test failed")
    }
}

this is memo:

package main

func main() {
    var a [10000][]int
    for i := 0; i < 1000; i++ {
        a[i] = make([]int, 1024)
    }
}

I make a large amount of memory and set only 10 bytes for the memory , but it won't signal segment fault signal any way.

Answer 1

RLIMIT_DATA describes the maximum size of a processes data segment. Traditionally, programs that allocate memory enlarge the data segment with calls to brk() to allocate memory from the operating system.

Go doesn't use this approach. Instead, it uses a variant of the mmap() system call to request regions of memory anywhere in the address space. This is much more flexible than a brk() based approach as you can deallocate arbitrary memory regions with munmap() , whereas a brk() based approach can only deallocate memory from the end of the data segment.

The result of this is that RLIMIT_DATA is inneffective in controlling the amount of memory a process uses. Try using RLIMIT_AS instead, but beware that this limit also incorporates the address space you use for file mappings, especially in the case of shared libraries.

Answer 2

There is one proposal Soft memory limit , it may be released after go 1.18

This option comes in two flavors: a new runtime/debug function called SetMemoryLimit and a GOMEMLIMIT environment variable. In sum, the runtime will try to maintain this memory limit by limiting the size of the heap, and by returning memory to the underlying platform more aggressively. This includes with a mechanism to help mitigate garbage collection death spirals. Finally, by setting GOGC=off, the Go runtime will always grow the heap to the full memory limit.

This new option gives applications better control over their resource economy. It empowers users to:

• Better utilize the memory that they already have,
• Confidently decrease their memory limits, knowing Go will respect them,
• Avoid unsupported forms of garbage collection tuning.

---

Update

This feature will be released in Go 1.19

The runtime now includes support for a soft memory limit. This memory limit includes the Go heap and all other memory managed by the runtime, and excludes external memory sources such as mappings of the binary itself, memory managed in other languages, and memory held by the operating system on behalf of the Go program.

This limit may be managed via runtime/debug.SetMemoryLimit or the equivalent GOMEMLIMIT environment variable.

The limit works in conjunction with runtime/debug.SetGCPercent / GOGC , and will be respected even if GOGC=off , allowing Go programs to always make maximal use of their memory limit, improving resource efficiency in some cases.