In malloc, why use brk at all? Why not just use mmap?

Question

Typical implementations of malloc use brk / sbrk as the primary means of claiming memory from the OS. However, they also use mmap to get chunks for large allocations. Is there a real benefit to using brk instead of mmap , or is it just tradition? Wouldn't it work just as well to do it all with mmap ?

(Note: I use sbrk and brk interchangeably here because they are interfaces to the same Linux system call, brk .)

---

For reference, here are a couple of documents describing the glibc malloc :

GNU C Library Reference Manual: The GNU Allocator
https://www.gnu.org/software/libc/manual/html_node/The-GNU-Allocator.html

glibc wiki: Overview of Malloc
https://sourceware.org/glibc/wiki/MallocInternals

What these documents describe is that sbrk is used to claim a primary arena for small allocations, mmap is used to claim secondary arenas, and mmap is also used to claim space for large objects ("much larger than a page").

The use of both the application heap (claimed with sbrk ) and mmap introduces some additional complexity that might be unnecessary:

Allocated Arena - the main arena uses the application's heap. Other arenas use mmap 'd heaps. To map a chunk to a heap, you need to know which case applies. If this bit is 0, the chunk comes from the main arena and the main heap. If this bit is 1, the chunk comes from mmap 'd memory and the location of the heap can be computed from the chunk's address.

[Glibc malloc is derived from ptmalloc , which was derived from dlmalloc , which was started in 1987.]

---

The jemalloc manpage ( http://jemalloc.net/jemalloc.3.html ) has this to say:

Traditionally, allocators have used sbrk(2) to obtain memory, which is suboptimal for several reasons, including race conditions, increased fragmentation, and artificial limitations on maximum usable memory. If sbrk(2) is supported by the operating system, this allocator uses both mmap(2) and sbrk(2), in that order of preference; otherwise only mmap(2) is used.

So, they even say here that sbrk is suboptimal but they use it anyway, even though they've already gone to the trouble of writing their code so that it works without it.

[Writing of jemalloc started in 2005.]

UPDATE: Thinking about this more, that bit about "in order of preference" gives me a line on inquiry. Why the order of preference? Are they just using sbrk as a fallback in case mmap is not supported (or lacks necessary features), or is it possible for the process to get into some state where it can use sbrk but not mmap ? I'll look at their code and see if I can figure out what it's doing.

---

I'm asking because I'm implementing a garbage collection system in C, and so far I see no reason to use anything besides mmap . I'm wondering if there's something I'm missing, though.

(In my case I have an additional reason to avoid brk , which is that I might need to use malloc at some point.)

Answer 1

The system call brk() has the advantage of having only a single data item to track memory use, which happily is also directly related to the total size of the heap.

This has been in the exact same form since 1975's Unix V6. Mind you, V6 supported a user address space of 65,535 bytes. So there wasn't a lot of thought given for managing much more than 64K, certainly not terabytes.

Using mmap seems reasonable until I start wondering how altered or added-on garbage collection could use mmap but without rewriting the allocation algorithm too.

Will that work nicely with realloc() , fork() , etc.?

Answer 2

Calling mmap(2) once per memory allocation is not a viable approach for a general purpose memory allocator because the allocation granularity (the smallest individual unit which may be allocated at a time) for mmap(2) is PAGESIZE (usually 4096 bytes), and because it requires a slow and complicated syscall. The allocator fast path for small allocations with low fragmentation should require no syscalls.

So regardless what strategy you use, you still need to support multiple of what glibc calls memory arenas, and the GNU manual mentions: "The presence of multiple arenas allows multiple threads to allocate memory simultaneously in separate arenas, thus improving performance."

---

The jemalloc manpage ( http://jemalloc.net/jemalloc.3.html ) has this to say:

Traditionally, allocators have used sbrk(2) to obtain memory, which is suboptimal for several reasons, including race conditions, increased fragmentation, and artificial limitations on maximum usable memory. If sbrk(2) is supported by the operating system, this allocator uses both mmap(2) and sbrk(2), in that order of preference; otherwise only mmap(2) is used.

I don't see how any of these apply to the modern use of sbrk(2) , as I understand it. Race conditions are handled by threading primitives. Fragmentation is handled just as would be done with memory arenas allocated by mmap(2) . The maximum usable memory is irrelevant, because mmap(2) should be used for any large allocation to reduce fragmentation and to release memory back to the operating system immediately on free(3) .

---

The use of both the application heap (claimed with sbrk) and mmap introduces some additional complexity that might be unnecessary:

Allocated Arena - the main arena uses the application's heap. Other arenas use mmap'd heaps. To map a chunk to a heap, you need to know which case applies. If this bit is 0, the chunk comes from the main arena and the main heap. If this bit is 1, the chunk comes from mmap'd memory and the location of the heap can be computed from the chunk's address.

So the question now is, if we're already using mmap(2) , why not just allocate an arena at process start with mmap(2) instead of using sbrk(2) ? Especially so if, as quoted, it is necessary to track which allocation type was used. There are several reasons:

1. mmap(2) may not be supported.
2. sbrk(2) is already initialized for a process, whereas mmap(2) would introduce additional requirements.
3. As glibc wiki says, "If the request is large enough, mmap() is used to request memory directly from the operating system [...] and there may be a limit to how many such mappings there can be at one time. "
4. A memory map allocated with mmap(2) cannot be extended as easily. Linux has mremap(2) , but its use limits the allocator to kernels which support it. Premapping many pages with PROT_NONE access uses too much virtual memory. Using MMAP_FIXED unmaps any mapping which may have been there before without warning. sbrk(2) has none of these problems, and is explicitly designed to allow for extending its memory safely.

Answer 3

mmap() didn't exist in the early versions of Unix. brk() was the only way to increase the size of the data segment of the process at that time. The first version of Unix with mmap() was SunOS in the mid 80's, the first open-source version was BSD-Reno in 1990.

And to be usable for malloc() you don't want to require a real file to back up the memory. In 1988 SunOS implemented /dev/zero for this purpose, and in the 1990's HP-UX implemented the MAP_ANONYMOUS flag.

There are now versions of mmap() that offer a variety of methods to allocate the heap.

Answer 4

The obvious advantage is that you can grow the last allocation in place , which is something you can't do with mmap(2) ( mremap(2) is a Linux extension, not portable).

For naive (and not-so-naive) programs which are using realloc(3) eg. to append to a string, this translates in a 1 or 2 orders of magnitude speed boost;-)

Answer 5

I don't know the details on Linux specifically, but on FreeBSD for several years now mmap is preferred and jemalloc in FreeBSD's libc has sbrk() completely disabled. brk()/sbrk() are not implemented in the kernel on the newer ports to aarch64 and risc-v.

If I understand the history of jemalloc correctly, it was originally the new allocator in FreeBSD's libc before it was broken out and made portable. Now FreeBSD is a downstream consumer of jemalloc. Its very possible that its preference for mmap() over sbrk() originated with the characteristics of the FreeBSD VM system that was built around implementing the mmap interface.

It's worth noting that in SUS and POSIX brk/sbrk are deprecated and should be considered non-portable at this point . If you are working on a new allocator you probably don't want to depend on them.