如何避免Java中的Java堆空间异常

Question

I'm reading data from IO having huge volume of data and I need to store the data in key value pair in Map or properties file, then only I can use that data for generating reports. But when I am storing this huge data in Map or Properties file, Heap Memory Exception is coming.Instead, if I am using SQLLite its taking very huge time to retrieve that. Is there any different way available to achieve this.Please suggest.

Answer 1

Java Heap Space Important Points

1. Java Heap Memory is part of memory allocated to JVM by Operating System.

2. Whenever we create objects they are created inside Heap in Java.

3. Java Heap space is divided into three regions or generation for sake of garbage collection called New Generation, Old or tenured Generation or Perm Space. Permanent generation is garbage collected during full gc in hotspot JVM.

4. You can increase or change size of Java Heap space by using JVM command line option -Xms, -Xmx and -Xmn. don't forget to add word "M" or "G" after specifying size to indicate Mega or Gig. For example you can set java heap size to 258MB by executing following command java -Xmx256m javaClassName (your program class name).

5. You can use either JConsole or Runtime.maxMemory(), Runtime.totalMemory(), Runtime.freeMemory() to query about Heap size programmatic in Java.

6. You can use command "jmap" to take Heap dump in Java and "jhat" to analyze that heap dump.

7. Java Heap space is different than Stack which is used to store call hierarchy and local variables.

8. Java Garbage collector is responsible for reclaiming memory from dead object and returning to Java Heap space.

9. Don't panic when you get java.lang.OutOfMemoryError, sometimes it's just matter of increasing heap size but if it's recurrent then look for memory leak in Java.

10. Use Profiler and Heap dump Analyzer tool to understand Java Heap space and how much memory is allocated to each object.

Reference link for more details:

https://docs.oracle.com/cd/E19159-01/819-3681/abeii/index.html

https://docs.oracle.com/cd/E40520_01/integrator.311/integratoretl_users/src/ti_troubleshoot_memory_errors.html

Answer 2

You need to do a rough estimate of of memory needed for your map. How many keys and values? How large are keys and values? For example, if the keys are longs and values are strings 40 characters long on average, the absolute minimum for 2 billion key-value pairs is (40 + 8) * 2E9 - approximately 100 GB. Of course, the real requirement is larger than the minimum estimate - as much as two times larger depending on the nature of the keys and values.

If the estimated amount of memory beyond reasonable (100 GB is beyond reasonable unless you have lots of money) you need to figure out a way to partition your processing. You need to read in a large chunk of data, then run some algorithm on it to reduce it to some small size. Then do it for all other chunks one by one, making sure to not keeping the old chunk around when you process the new chunk. Finally, look at the results from all chunks and compute the final result. For a better description of this approach, look up "map-reduce'.

If the estimated amount of memory is somewhat reasonable (say, 8 GB - and you have a 16 GB machine) - use 64 bit JVM, set the maximum heap memory using -Xmx switch, make sure you use the most efficient data structures such as Trove maps.

Good luck!

Answer 3

增加堆大小是一种选择，但有通过使用java.You存储器映射文件可以引用存储断堆数据替代该