Memory 没有释放 std::list <std::shared_ptr<std::string> &gt; C++ </std::shared_ptr<std::string>

Question

I'm populating a list of string shared pointers. At some point in my program, I clear the list. But the memory consumption of my program does not reduce even after I call clear() function of list. Any Idea why?

#include <memory>
#include <string>
#include <list>
#include <iostream>

int main()
{
    std::string text = "                                                            \
        PREFACE                                                                     \
                                                                                    \
        Most of the adventures recorded in this book really occurred; one or two    \
        were experiences of my own, the rest those of boys who were schoolmates     \
        of mine. Huck Finn is drawn from life; Tom Sawyer also, but not from an     \
        individual--he is a combination of the characteristics of three boys whom   \
        I knew, and therefore belongs to the composite order of architecture.       \
                                                                                    \
        The odd superstitions touched upon were all prevalent among children and    \
        slaves in the West at the period of this story--that is to say, thirty or   \
        forty years ago.                                                            \
                                                                                    \
        Although my book is intended mainly for the entertainment of boys and       \
        girls, I hope it will not be shunned by men and women on that account,      \
        for part of my plan has been to try to pleasantly remind adults of what     \
        they once were themselves, and of how they felt and thought and talked,     \
        and what queer enterprises they sometimes engaged in.                       \
    ";

    std::list<std::shared_ptr<std::string>> data;
    for (auto i = 0u; i < 999999; ++i) {
        data.push_back(std::make_shared<std::string>(text));
    }

    std::cout << "Data loaded. Press any key to continue...";
    std::cin.get();

    data.clear(); // memory does not reduce
    std::cout << "Data unloaded. Press any key to continue...";
    std::cin.get();

    std::cout << "Container size:" << data.size() << std::endl;
    std::cout << "Press any key to exit...";
    std::cin.get();

    return 0;
}

I'm debugging on WSL. I used both linux top (on WSL) and Windows Task Manager to check memory usage at each stop. But valgrind does not report any memory leaks.

PS Please don't ask why I'm using shared pointers. Proposing different approach would be useful but the main purpose of this question is to understand this behavior. Since even cppref doesn't explain this. Appreciate if someone can explain this behavior rather than fixing it.

I don't need a fix. I need an explanation.

Answer 1

I used both linux top (on WSL) and Windows Task Manager to check memory usage at each stop. But valgrind does not report any memory leaks.

TLDR : Allocating memory is an expensive operation relative to other code paths. To improve performance, almost all heap managers will continue to hold onto the memory for subsequent allocations instead of giving it directly back from where it came from.

When memory is allocated by your program code, it goes through a tier of memory heaps to grant that allocation. When your program invokes "new" (via make_shared ), it calls into the C/C++ runtime to allocate memory. The C/C++ runtime, if it doesn't have a sufficiently large enough contiguous byte range in its heap to grant that allocation, it will thunk down to the process heap via OS specific library calls to ask for more memory. The process heap, if it doesn't have enough to allocate right away, it makes a system call to allocate more virtual memory... and probably a few more heaps as well. And let's not forget that memory likely has to be paged in, but I digress.

Each one of these heap accesses requires taking a lock on a data structure to manage the heap allocation being requested, and possibly a system call to the OS. And possibly some extra effort to collapse or re-arrange blocks of memory as needed. That's why memory heaps are tiered. If every new and delete call were to go directly to the virtual memory manager, program and system performance would be really slow from the sheer number of system calls to do this.

Similarly releasing memory back to where it came from is also a similar performance hit. It's possible that these heaps will release back to its parent heap when it wants to compact, but don't expect it to do that from a single invocation of "delete".

Tools like Top and Task Manager can only observe the amount of virtual memory allocated by the process from the OS. They aren't aware of free'd allocations being managed by the runtime libraries of your program. Whereas Valgrind instruments itself into your code and can hook itself closer to the C++ memory manager. But even Valgrind will report false positives every now and then.

Answer 2

As Nicolai Josuttis mentioned in this talk , make_shared does not call new twice, instead it allocates memory just one both for control block and for the resource you are pointing to, and so it won't deallocate the block of the memory until all the shared pointers and weak pointers that are referencing to that control block are deleted

however in this case no more shared/weak pointers are pointing to those control blocks, infact, if instead of using std::string you use a your object, you will see the destructors been called ad the clear() call:

#include <memory>
#include <string>
#include <list>
#include <iostream>
class A{
public:
    ~A(){std::cout<<"destructor"<<std::endl;}
};
int main()
{

    std::list<std::shared_ptr<A>> data;
    for (auto i = 0u; i < 10; ++i) {
        data.push_back(std::make_shared<A>());
    }

    std::cout << "Data loaded. Press any key to continue...";
    std::cin.get();

    data.clear(); // memory does not reduce
    std::cout << "Data unloaded. Press any key to continue...";
    std::cin.get();

    std::cout << "Container size:" << data.size() << std::endl;
    std::cout << "Press any key to exit...";
    std::cin.get();

    return 0;
}

OUTPUT:

Data loaded. Press any key to continue...
destructor
destructor
destructor
destructor
destructor
destructor
destructor
destructor
destructor
destructor
Data unloaded. Press any key to continue...

So you can't see the memory decreasing immediately because the allocator that you are using is optimizing the memory, and so freeing memory will only means "free to overwrite this piece of memory" and not giving it back to the OS, in order to use that in future allocation, without involving every time the OS

Answer 3

default use tcmalloc, you can use jemalloc with "export LD_PRELOAD=/usr/lib64/libjemalloc.so.1". The tcmalloc allocator optimized the memory used, it suppose the prog will use memory next time for lots of min memory, if you use vector( continuous memory), the tcmalloc will free it after destruct it. or you can add some code after list used like:

std::vector<std::string> myVec;
for(int i = 0; i < 1000; ++i)
{
    myVec.push_back(string(256, 'a'));
}
myVec.clear();

Then you'll find: the list memory is freed!