malloc consumption much less than process itself

Question

I am developing hash table in C using gcc.

for collisions I am using linked list.

when I load ~1 GB file, mallinfo() show:

Total non-mmapped bytes        arena      =      1753260032
# of free chunks               ordblks    =               2
# of free fastbin blocks       smblks     =               0
# of mapped regions            hblks      =               1
Bytes in mapped regions        hblkhd     =         8003584
Max. total allocated space     usmblks    =               0
Free bytes held in fastbins    fsmblks    =               0
Total allocated space          uordblks   =      1753126384
Total free space               fordblks   =          133648
Topmost releasable block       keepcost   =          133072

if I calculate using Excel the memory consumption I get around 2 GB, which is similar to 1.7 GB shown by mallinfo().

at the same time, the linux process shown in top or htop uses 5.7 GB.

tried jemalloc and tcmalloc, results are more or less similar. process always uses around 5.7 GB.

I know malloc must do some linked list of free memory "pieces" or something similar, but 5 GB is way more than 2 GB, even if memory blocks are aligned.

where the wasted memory go? :-)

Answer 1

In this case mallinfo() gives false information.

When I add mallinfo() after every 1M allocations, i can clearly see how int is overflown, became negative, then overlflow again and became positive again.

Excel calculation seems not correct too. I added code for calculation of allocated memory, and result is about 4.7 GB.

There are still 1 GB difference between what top shown and real consumption, but this time it could be easily explain by allocator's internal data