memory leak disappears when using valgrind

Question

We have a complex algorithm, which processes OpenCV images, thereby allocating and deallocating several GB of memory, mostly cv::Mat with about 10MB size each. If we run this iteratively unter valgrind (either with --tool=massif or --tool=memcheck) the memory footprint returns to the same value (+-1MB) after each iteration, and no significant memory leak is found. Watching from outside via ps or pmap or from inside via /proc/self/status also shows a maximum footprint of 2.3GB not increasing.

If we run the same software without valgrind however, then the memory footprint (checked from outside via ps or pmap or from inside via /proc/self/status) increases with every iteration about several hundred MB, soon reaching 5BG after a few iterations.

Thus we have something looking like a memory leak, but valgrind is of no help for finding the cause. What could this be?

(This is C++ under Ubuntu).

Answer 1

Thanks to the comment of @phd I found a solution to my problem: Using tcmalloc reduces the memory footprint dramatically (using 2.5GB instead of 6GB). See the attached grafic

RSS memory using different malloc libraries

There seems to be still a slight increase in memory usage with tcmalloc or jemalloc, but at least it's ok for the iteration numbers we usally have.

Still wonder though how malloc can waste so much resources. I tried to find out with malloc_info(), but with no success. I suspect memory fragmentation and/or multithreading plays a role here.