Abort in glibc while trying to use sbrk to reduce the size of the data segment

Question

While working with glibc I tried to reduce the data segment using sbrk using a negative parameter, and found a most strange behaviour.

I first malloc , then free it, then reduce data segment with sbrk , and then malloc again with same size as the first one.

The issue is, if the malloc size (both malloc s with same size) is small enough (32k, or eight 4k pages) then everything works fine. But when I increase a little the malloc - free - malloc size (to nine 4k pages) then I get the core dump. What is even more strange is that when I raise the malloc size to cross the mmap threshold (128k) then I get the adjust-abort behaviour.

The C code:

#define _GNU_SOURCE 1
#include <stdio.h>
#include <malloc.h>
#include <unistd.h>

// set MMAP_ALLOC_SIZE to 8 4k-pages it will work, 
// set it to 9 4k-pages it raises a 'segmentation fault (core dumped)'
// set it to 33 4k-pages it raises a 'break adjusted to free malloc space' and 'abort (core dumped)'
#define MMAP_ALLOC_SIZE   (33 * 4096)
#define PRINT_MEM { \
    struct mallinfo mi; \
    mi = mallinfo(); \
    printf("ptr    %p\n", ptr); \
    printf("brk(0) %p\n", sbrk(0)); \
    printf("heap   %d bytes\n", mi.arena); \
    printf("mmap   %d bytes\n\n", mi.hblkhd); \
}

int main(int argc, char *argv[])
{
    void *ptr;
    ptr = NULL;                      PRINT_MEM
    printf("1) will malloc > MMAP_THRESHOLD (128 KiB) ...\n");
    ptr = malloc(MMAP_ALLOC_SIZE);   PRINT_MEM
    printf("2) will free malloc ...\n");
    free(ptr);                       PRINT_MEM
    printf("3) will reduce brk  ...\n");
    ptr = sbrk(-100000);             PRINT_MEM
    printf("4) will malloc > MMAP_THRESHOLD (128 KiB) ... \n");
    ptr = malloc(MMAP_ALLOC_SIZE);   PRINT_MEM
    printf("5) completion.\n"); // never happens if MMAP_ALLOC_SIZE is > 8 4k-pages
    return 0;
}

Compiled with:

gcc -Wall testbrk.c -o testbrk

Which gives the successful output for MMAP_ALLOC_SIZE (8 * 4096) :

ptr    (nil)
brk(0) 0xf46000
heap   0 bytes
mmap   0 bytes

1) will malloc > MMAP_THRESHOLD (128 KiB) ...
ptr    0xf25670
brk(0) 0xf46000
heap   135168 bytes
mmap   0 bytes

2) will free malloc ...
ptr    0xf25670
brk(0) 0xf46000
heap   135168 bytes
mmap   0 bytes

3) will reduce brk  ...
ptr    0xf46000
brk(0) 0xf2d960
heap   135168 bytes
mmap   0 bytes

4) will malloc > MMAP_THRESHOLD (128 KiB) ... 
ptr    0xf25670
brk(0) 0xf2d960
heap   135168 bytes
mmap   0 bytes

5) completion.

The following abort output for MMAP_ALLOC_SIZE (9 * 4096) :

ptr    (nil)
brk(0) 0x1b7f000
heap   0 bytes
mmap   0 bytes

1) will malloc > MMAP_THRESHOLD (128 KiB) ...
ptr    0x1b5e670
brk(0) 0x1b7f000
heap   135168 bytes
mmap   0 bytes

2) will free malloc ...
ptr    0x1b5e670
brk(0) 0x1b7f000
heap   135168 bytes
mmap   0 bytes

3) will reduce brk  ...
ptr    0x1b7f000
brk(0) 0x1b66960
heap   135168 bytes
mmap   0 bytes

4) will malloc > MMAP_THRESHOLD (128 KiB) ... 
Segmentation fault (core dumped)

And the following adjust-abort output for MMAP_ALLOC_SIZE (33 * 4096) :

ptr    (nil)
brk(0) 0x1093000
heap   0 bytes
mmap   0 bytes

1) will malloc > MMAP_THRESHOLD (128 KiB) ...
ptr    0x7fdd1c7f6010
brk(0) 0x1093000
heap   135168 bytes
mmap   139264 bytes

2) will free malloc ...
ptr    0x7fdd1c7f6010
brk(0) 0x1093000
heap   135168 bytes
mmap   0 bytes

3) will reduce brk  ...
ptr    0x1093000
brk(0) 0x107a960
heap   135168 bytes
mmap   0 bytes

4) will malloc > MMAP_THRESHOLD (128 KiB) ... 
break adjusted to free malloc space
Aborted (core dumped)

So the sbrk reduction call works without error, but the subsequent malloc raises a core dump even if enough memory was still available.

I'm I doing something wrong or is this a limitation in the data segment resize?

Answer 1

It is well-documented that glibc malloc uses sbrk internally. Absent a statement that says otherwise, it can also use memory obtained with sbrk for internal bookkeeping purposes. It is neither documented nor guessable where exactly this internal bookkeeping data is stored. Thus, taking away any memory obtained by malloc (via sbrk or otherwise) can invalidate this data.

It follows that sbrk with a negative argument should never be used in a program that also uses malloc (and of course any library function that might use malloc , such as printf ). A statement to this effect probably should have been included in the glibc documentation, to make the reasoning above unnecessary. There is a statement that cautions against the use of brk and sbrk in general though:

You will not normally use the functions in this section, because the functions described in Memory Allocation are easier to use. Those are interfaces to a GNU C Library memory allocator that uses the functions below itself. The functions below are simple interfaces to system calls.

If you want to release unused memory at the end of the glibc malloc arena, use malloc_trim() (a glibc extension, not a standard C or POSIX function).

Answer 2

As suggested by the accepted answer, malloc_trim does the job perfectly. Change the main to:

int main(int argc, char *argv[])
{
    void *ptr;
    ptr = NULL;                      PRINT_MEM
    printf("1) will malloc > MMAP_THRESHOLD (128 KiB) ...\n");
    ptr = malloc(MMAP_ALLOC_SIZE);   PRINT_MEM
    printf("2) will free malloc ...\n");
    free(ptr);                       PRINT_MEM
    printf("3) will reduce malloc_trim  ...\n");
    int r = malloc_trim(0);             PRINT_MEM
    printf("r is %d\n", r);
    printf("4) will malloc > MMAP_THRESHOLD (128 KiB) ... \n");
    ptr = malloc(MMAP_ALLOC_SIZE);   PRINT_MEM
    printf("5) completion.\n"); // never happens if MMAP_ALLOC_SIZE is > 8 4k-pages
    return 0;
}

Which gives the successful output for MMAP_ALLOC_SIZE (8 * 4096):

ptr    (nil)
brk(0) 0x4b3000
heap   0 bytes
mmap   0 bytes

1) will malloc > MMAP_THRESHOLD (128 KiB) ...
ptr    0x492670
brk(0) 0x4b3000
heap   135168 bytes
mmap   0 bytes

2) will free malloc ...
ptr    0x492670
brk(0) 0x4b3000
heap   135168 bytes
mmap   0 bytes

3) will reduce malloc_trim  ...
ptr    0x492670
brk(0) 0x493000
heap   4096 bytes
mmap   0 bytes

r is 1
4) will malloc > MMAP_THRESHOLD (128 KiB) ... 
ptr    0x492670
brk(0) 0x4bb000
heap   167936 bytes
mmap   0 bytes

5) completion.

The following successful output for MMAP_ALLOC_SIZE (9 * 4096):

ptr    (nil)
brk(0) 0xe30000
heap   0 bytes
mmap   0 bytes

1) will malloc > MMAP_THRESHOLD (128 KiB) ...
ptr    0xe0f670
brk(0) 0xe30000
heap   135168 bytes
mmap   0 bytes

2) will free malloc ...
ptr    0xe0f670
brk(0) 0xe30000
heap   135168 bytes
mmap   0 bytes

3) will reduce malloc_trim  ...
ptr    0xe0f670
brk(0) 0xe10000
heap   4096 bytes
mmap   0 bytes

r is 1
4) will malloc > MMAP_THRESHOLD (128 KiB) ... 
ptr    0xe0f670
brk(0) 0xe39000
heap   172032 bytes
mmap   0 bytes

5) completion.

And the following successful output for MMAP_ALLOC_SIZE (33 * 4096):

ptr    (nil)
brk(0) 0xa5b000
heap   0 bytes
mmap   0 bytes
    
1) will malloc > MMAP_THRESHOLD (128 KiB) ...
ptr    0x7fd54f4c3010
brk(0) 0xa5b000
heap   135168 bytes
mmap   139264 bytes

2) will free malloc ...
ptr    0x7fd54f4c3010
brk(0) 0xa5b000
heap   135168 bytes
mmap   0 bytes

3) will reduce malloc_trim  ...
ptr    0x7fd54f4c3010
brk(0) 0xa3b000
heap   4096 bytes
mmap   0 bytes

r is 1
4) will malloc > MMAP_THRESHOLD (128 KiB) ... 
ptr    0xa3a670
brk(0) 0xa7c000
heap   270336 bytes
mmap   0 bytes

5) completion.

And issue solved!

Note also that heap size from mallinfo is now being updated correctly using malloc_trim with zero argument (which leaves a 4k page at the top of the arena), while using sbrk with negative argument does move brk(0) down, but leaves mallinfo heap size untouched.

Thanks a lot: (still filling that bug report to glibc developers though :)