收藏和记忆

Question

I have an application that read 3-4 GB s of data, build entities out of each line and then stores them in Lists.

The problem I had is, memory grows insane becomes like 13 to 15 GB. Why the heck storing these entities takes so much memory.

So I build a Tree and did something similar to Huffman Encoding, and overall memory size became around 200 - 300 MB.

I understand, that I compacted the data. But I wasn't expecting that storing objects in the list would increase the memory so much. Why did that happen?

how about other data structures like dictionary, stack, queue, array etc?

Where can I find more information about the internals and memory allocations of data structures?

Or am I doing something wrong?

Answer 1

In .NET large objects go on the large object heap which is not compacted. Large is everything above 85,000 bytes. When you grow your lists they will probably become larger than that and have to be reallocated once you cross the current capacity. Rellocation means that they are very likely put at the end of the heap. So you end up with a very fragmented LOH and lots of memory usage.

Update : If you initialize your lists with the required capacity (which you can determine from the DB I guess) then your memory consumption should go down a bit.

Answer 2

Regardless of the data structure you're going to use, your memory consumption is never going to drop below the memory required to store all your data.

Have you calculated how much memory it is required to store one instance class object?

Your huffman encoding is a space-saving optimization, which means that you are eliminating a lot of duplicated data within your class objects yourself. This has nothing to do with the data structure you use to hold your data. This depends on how your data itself is structured so that you can take advantage of different space-saving strategies (of which huffman encoding is one out of many possibilities, suitable for eliminating common prefixes and the data structure used to store it is a tree).

Now, back to your question. Without optimizing your data (ie objects), there are things you can watch out to improve memory usage efficiency.

Are all our objects of similar size?

Did you simply run a loop, allocate memory on-the-fly, then insert them into a list, like this:

foreach (var obj in collection) { myList.Add(new myObject(obj)); }

In that case, your list object is constantly being expanded. And if there is not enough free memory at the end to expand the list, .NET will allocate a new, larger piece of memory and copies the original array to the new memory. Essentially you end up with two pieces of memory -- the original one, and the new expanded one (now holding the list). Do this many many many times (as you obviously need to for GB's of data), and you are looking at a LOT of fragmented memory spaces.

You'll be better off just allocating enough memory for the entire list at one go.

As an afternote, I can't help but wondering: how in the world are you going to search this HUGE list to find something you need? Shouldn't you be using something like a binary tree or a hash-table to aid in your searching? Maybe you are just reading in all the data, perform some processing on all of them, then writing them back out...

Answer 3

If you are using classes, read the response of this: Understanding CLR object size between 32 bit vs 64 bit

On 64 bits (you are using 64 bits, right?) object overhead is 16 bytes PLUS the reference to the object (someone is referencing him, right?) so another 8 bytes. So an empty object will "eat" at least 24 bytes.

If you are using List s, remember that List s grow by doubling, so you could be wasting much space. Other .NET collections grow in the same way.

I'll add that the "pure" overhead of million of List s could bring the memory to his knees. Other than the 16 + 8 bytes of space "eaten" by the List object, it is composed (in the .NET implementation) of 2 ints (8 bytes), a SyncLock reference (8 bytes, it's null normally) and a reference to the internal array (so 8 + 16 bytes + the array)