Linux Allocator Does Not Release Small Chunks of Memory

Question

The Linux glibc allocator seems to be behaving weirdly. Hopefully, someone can shed some light on this. Here is the source file that I have:

first.cpp:

#include <unistd.h>
#include <stdlib.h>
#include <list>
#include <vector>

int main() {

  std::list<char*> ptrs;
  for(size_t i = 0; i < 50000; ++i) {
    ptrs.push_back( new char[1024] );
  }
  for(size_t i = 0; i < 50000; ++i) {
    delete[] ptrs.back();
    ptrs.pop_back();
  }

  ptrs.clear();

  sleep(100);

  return 0;
}

second.cpp:

#include <unistd.h>
#include <stdlib.h>
#include <list>

int main() {

  char** ptrs = new char*[50000];
  for(size_t i = 0; i < 50000; ++i) {
    ptrs[i] = new char[1024];
  }
  for(size_t i = 0; i < 50000; ++i) {
    delete[] ptrs[i];
  }
  delete[] ptrs;

  sleep(100);

  return 0;
}

I compile both:

$ g++ -o first first.cpp
$ g++ -o second second.cpp

I run first, and after it's sleeping, I see the resident memory size:

When I compile first.cpp, and run it, I look at memory with ps:

$ ./first&
$ ps aux | grep first
davidw    9393  1.3  0.3  64344 53016 pts/4    S    23:37   0:00 ./first


$ ./second&
$ ps aux | grep second
davidw    9404  1.0  0.0  12068  1024 pts/4    S    23:38   0:00 ./second

Notice the resident memory size. In first, the resident memory size is 53016k. in second, it is 1024k. First never released the allocations back to the kernel for some reason or another.

Why does the first program not relinquish memory to the kernel, but the second program does? I understand that the first program uses a linked list and the linked list probably allocates some nodes on the same page as the data we're freeing. However, those nodes should be freed, as we're popping those nodes off, then clearing the linked list. If you run either of these programs through valgrind, it comes back with no memory leaks. What is probably happening is memory gets fragmented in first.cpp that doesn't in second.cpp. However, if all memory on a page is freed, how does that page not get relinquished back to the kernel? What does it take for memory to get relinquished back to the kernel? How can I modify first.cpp (continuing to put the char*'s in a list) so that the memory is relinquished to the kernel.

Answer 1

This behaviour is intentional, there is a tunable threshold that glibc uses to decide whether to actually return memory to the system or whether to cache it for later reuse. In your first program you make lots of small allocations with each push_back and those small allocations are not a contiguous block and are presumably below the threshold, so don't get returned to the OS.

Calling malloc_trim(0) after clearing the list should cause glibc to immediately return the top-most region of free memory to the system (requiring a sbrk system call next time memory is needed.)

If you really need to override the default behaviour (which I wouldn't recommend unless profiling reveals it actually helps) then you should probably use strace and/or experiment with mallinfo to see what's actually happening in your program, and maybe using mallopt to adjust the threshold for returning memory to the system.

Answer 2

It keeps the smaller chunks available in case you request them again. It is a simple caching optimization, and not behaviour to be concerned about.

Answer 3

Typically, the memory allocated by new will only be returned to the system when the process terminates. In the second case, I suspect that libc is using a special allocator for very large continuous blocks, which does return it, but I'd be very surprised if any of your new char[1024] were returned, and on many Unices, even the large block won't be returned.

Answer 4

(Editing down my answer, since there really isn't any issue here.)

As has been noted, there isn't really an issue here. Johnathon Wakely hits the nail on the head.

When the memory utilization is not what I expect it to be on Linux, I usually start my analysis using the mtrace tool, and analyzing the /proc/self/maps file.

mtrace is used by bracketing your code around two calls, one to starts the trace, and one that ends it.

  mtrace();
  {
      // do stuff
  }
  muntrace();

The mtrace calls are only active if the MALLOC_TRACE environment variable is set. It specifies the name of the file for the mtrace logging output. This logging output can then be analyzed for memory leaks. A command line program called mtrace can be used to analyze the output.

$ MALLOC_TRACE=mtrace.log ./a.out
$ mtrace ./a.out mtrace.log

The /proc/self/maps file provides a list of memory mapped regions in use by the current program, including anonymous regions. It can help identify regions which are particularly large, and then additional sleuthing is needed to determine what that region is associated with. Below is a simple program to dump the /proc/self/maps file to another file.

void dump_maps (const char *outfilename) {
  std::ifstream inmaps("/proc/self/maps");
  std::ofstream outf(outfilename, std::ios::out|std::ios::trunc);
  outf << inmaps.rdbuf();
}