填充Hashmap时，GC Overhead Limit超出

Question

When my java program starts, it fills a hashmap with thousands of objects. The key is a string and the value is a set of objects.

The program is running at full tilt and throws a Out of Memory Exception: GC Overhead Limit Exceeded when it reaches around 10000 keys.

I have read that it could be that the underlying array has to constantly resize. But id like to be able to solve this without simply increasing the heap size.

Thanks!

Answer 1

You need to model the size of your data, plus the per-element overhead of HashMap , in order to determine your heap requirements.

For the sake of simplicity I'll assume you're running a 64-bit JVM using compressed OOPS (OOP = ordinary object pointer). This gives a 12-byte header per object, and a 4-byte object reference. I'll further assume you're using a HashMap with a default load factor of 0.75.

With 10,000 elements, the table size is a minimum of 10,000 / 0.75 = 13,333. However, the table size is always a power of two, so it'll probably be 16,384 long. That gives 65,536 bytes -- 64KB.

Each element stored in a HashMap also requires the creation of an internal Node object, which has four 4-byte fields (hash, key, value, next) plus 12 bytes for the object header, giving 28 bytes per Node object. With 10,000 elements, that's 280KB.

The HashMap table size plus the internal Node objects thus require 344KB of overhead for storing 10,000 key-value pairs. That's not causing you to run out of memory. Changing the initial capacity of the HashMap will reduce the copying overhead from resizing, but the amount of temporary extra memory it takes is negligible compared to a typical heap size of hundreds of MB or several GB.

If your heap size is 1 GB and you're running out of memory at 10,000 map entries, then each key-value pair takes about 100KB. You won't be able to load a few million keys unless you drastically increase your heap size, or you reduce the size of each key-value pair, or some combination of both.

Answer 2

You are geting Out of Memory because internal array is doubled each time when a certain threshold is met. So you need to make sure it won't be happening. If you know the number of objects you need to store in the map just use constructor HashMap(int initialCapacity, float loadFactor) and provide the expected size in the first parameter. If you don't know the number of objects you still can try to set initialCapacity so some approximate value and/or play with loadFactor parameter (default value is 0.75) - the larger the value the larger the threshold at wich there will be the resize.

Answer 3

You should use profilers and memory analyzer tools for performance and memory related issues. One of the tips I would do is to configure JVM with flag JAVA_OPTS=-XX:-HeapDumpOnOutOfMemoryError Then when you've got this exception and JVM crashes, you may take this dump and use MAT memory analyzer tool to analyze objects distribution and root paths. Also one would prefer to use Java Mission Control JMC distributed with Java and it's flight recorder. Please note that flight recorder is available since Java 7 update 40