hadoop reducer上的JVM崩溃

Question

I am running java codes on hadoop, but encounter this error:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f2ffe7e1904, pid=31718, tid=139843231057664
#
# JRE version: Java(TM) SE Runtime Environment (8.0_72-b15) (build 1.8.0_72-b15)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.72-b15 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x813904]  PhaseIdealLoop::build_loop_late_post(Node*)+0x144
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /hadoop/nm-local-dir/usercache/ihradmin/appcache/application_1479451766852_3736/container_1479451766852_3736_01_000144/hs_err_pid31718.log
#
# Compiler replay data is saved as:
# /hadoop/nm-local-dir/usercache/ihradmin/appcache/application_1479451766852_3736/container_1479451766852_3736_01_000144/replay_pid31718.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp

When I go to the node manager, all the logs are aggregated since yarn.log-aggregation-enable is true , and log hs_err_pid31718.log and replay_pid31718.log cannot be found.

Normally 1) the JVM crashes after several minutes of reducer, 2) sometimes the auto-retry of reducer can succeeds, 3) some reducers can succeed without failure.

Hadoop version is 2.6.0, Java is Java8. This is not a new environments, we have lots of jobs running on the cluster.

My questions:

1. Can I find hs_err_pid31718.log anywhere after yarn aggregate the log and remove the folder? Or is there a setting to keep all the local logs so I can check the hs_err_pid31718.log while aggregating logs by yarn?

2. What's the common steps to narrow down the deep dive scope? Since the jvm crashed, I cannot see any exception in code. I have tried -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp this args but there is no heap dumped on the host failing the reduce tasks.

Thanks for any suggestion.

Answer 1

Answers

1. Use -XX:ErrorFile=<your prefered location>/hs_err_pid<pid>.log to set the hs_error file location to your prefered one.
2. Crash is due to JDK bug JDK-6675699 this has already fixed in JDK9 and backports are available on JDK8 update 74 onwards.

You are using JDK8 update 72. Kindly upgrade to latest version from here to avoid this crash.