Remove unused allocated Memory from HashMaps

Question

I want to read some XML-files and convert it to a graph (no graphics, just a model). But because the files are very large (2,2 GB) my model object, which holds all the information, becomes even larger (4x the size of the file...).

Googling through the net I tried to find ways to reduce the object size. I tried different collection types but would like to stick to a HashMap (because I have to have random access). The actuall keys and values make up just a small amount of the allocated memory. Most of the hash table is empty...

If I'm not totally wrong a garbage collection doesn't help me to free the allocated memory and reduce the size of the hashmap. Is there and other way to release unused memory and shrink the hashmap? Or is there a way to do perfect hashing? Or shoud I just use another collection?

Thanks in advance,

Sebastian

Answer 1

A HashMap is typically just a large array of references filled to a certain percentage of capacity. If only 80% of the map is filled, the remaining 20% of the array cells are unused (ie, are null). The extra overhead is really only just the empty (null) cells.

On a 32-bit CPU, each array cell is usually 4 bytes in size (although some JVM implementations may allocate 8 bytes). That's not really that much unused space overall.

Once your map is filled, you can copy it to another HashMap with a more appropriate (smaller) size giving a larger fill percentage.

Your question seems to imply that there are more allocated but unused objects that you're worried about. But how is that the case?

Addendum

Once a map is filled almost to capacity (typically more than 95% or so), a larger array is allocated, the old array's contents are copied to the new array, and then the smaller array is left to be garbage collected. This is obviously an expensive operation, so choosing a reasonably large initial size for the map is key to improving performance.

If you can (over)estimate the number of cells needed, preallocating a map can reduce or even eliminate the resizing operations.

Answer 2

What you are asking is not so clear, it is not clear if memory is taken by the objects that you put inside the hasmap or by the hashmap itself, which shouldn't be the case since it only holds references.

In any case take a look at the WeakHashMap , maybe it is what you are looking for: it is an hashmap which doesn't guarantee that keys are kept inside it, it should be used as a sort of cache but from your description I don't really know if it is your case or not.

Answer 3

If you get nowhere with reducing the memory footprint of your hashmap, you could always put the data in a database. Depending on how the data is accessed, you might still get reasonable performance if you introduce a cache in front of the db.

Answer 4

One thing that might come into play is that you might have substrings that are referencing old larger strings, and those substrings are then making it impossible for the GC to collect the char arrays that are too big.

This happens when you are using some XML parsers that are returning attributes/values as substring from a larger string. (A substring is only a limited view of the larger string).

Try to put your strings in the map by doing something like this:

map.put(new String(key), new String(value));

Note that the GC then might get more work to do when you are populating the map, and this might not help you if you don't have that many substrings that are referencing larger strings.

Answer 5

If you're really serious about this and you have time to spare, you can make your own implementation of the Map interface based on minimal perfect hashing

If your keys are Strings, then there apparently is a map available for you here . I haven't tried it myself but it brags about reduced memory usage.

Answer 6

You might give the Trove collections a shot. They advertise it as a more time and space efficient drop-in replacement for the java.util Collections.