How to use malloc() allocates memory more than RAM in redhat?

Question

System information: Linux version 2.6.32-573.12.1.el6.x86_64 (mockbuild@x86-031.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Mon Nov 23 12:55:32 EST 2015

RAM 48 GB

Problem: I want to malloc() 100 GB memory. But it fail to allocate on redhat system.

I find that 100GB can be allocated in macOS with 8 GB RAM (clang compile). I am very confuse about that.

Maybe lazy allocation described in this link? Why malloc() doesn't stop on OS X?

But why linux system can not? I try ubuntu and redhat, both fail to to that.

---

Result : After investigation, I find that the following two steps will let malloc() unlimited:

1. echo "1" > /proc/sys/vm/overcommit_memory

2. ulimit -v unlimited

Answer 1

Possible reasons:

1. The system is out of physical RAM or swap space.

2. In 32 bit mode, the process size limit was hit.

Possible solutions:

1. Reduce memory load on the system.

2. Increase physical memory or swap space.

3. Check if swap backing store is full.

I think these point can help you to understand malloc failure issue.

Input validation. For example, you've asked for many gigabytes of memory in a single allocation. The exact limit (if any) differs by malloc implementation; POSIX says nothing about a maximum value, short of its type being a size_t.

Allocation failures. With a data segment-based malloc, this means brk failed. This occurs when the new data segment is invalid, the system is low on memory, or the process has exceeded its maximum data segment size (as specified by RLIMIT_DATA). With a mapping-based malloc, this means mmap failed. This occurs when the process is out of virtual memory (specified by RLIMIT_AS), has exceeded its limit on mappings, or has requested too large a mapping. Note most modern Unix systems use a combination of brk and mmap to implement malloc, choosing between the two based on the size of the requested allocation.