分析hadoop集群中被杀死的Java进程

Question

Java program is executing in Hadoop cluster. It got OutOfMemoryError during the execution and process stopped. I want to analyze the killed java process for memory & other details. Where i can find the killed process log files? I used sar utility to analysis the memory but it shows only system memory instead of process memory.

Answer 1

您可以使用-XX:ErrorFile=<your location>/hs_err_pid<pid>.log作为JVM参数来设置hs_error文件位置。

Answer 2

First, perhaps you JVM has not been configured with enough heap size for your application.

That said, I have a little recommendation based on my experience. I think you should enable these flags to investigate which objects are consuming too much space.

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=path

The first flag is going to dump the heap in case of OOM exception and the second one is going to set the path where the JVM saves that dump file.

After getting the heap dump file you should use Memory Analyzer Tool ( https://www.eclipse.org/mat/ ) to see possible memory leaks caused by applications.

In addition, it is important to measure the GC process and you can do this with these flags.

-XX:+PrintReferenceGC 
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps - 
XX:+PrintTenuringDistribution -XX:+PrintAdaptiveSizePolicy 
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -
Xloggc:/some/where/gc.log

The GC.log can be analyzed using this online tool

http://gceasy.io/