Hadoop-启动YARN服务时Java Runtime Environment的内存不足

Question

I have set up a cluster (1-master & 2-slaves(slave1, slave2)) based on the tutorial http://pingax.com/install-apache-hadoop-ubuntu-cluster-setup . When I ran for the first time both HDFS & YARN services ran without any problem. But when I stopped an ran them again, I got the following while running YARN service ( start-yarn.sh ) from the master.

# starting yarn daemons
# starting resourcemanager, logging to /local/hadoop/logs/yarn-dev-resourcemanager-login200.out
# 
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 168 bytes for AllocateHeap
# An error report file with more information is saved as: /local/hadoop/hs_err_pid21428.log

Compiler replay data is saved as: /local/hadoop/replay_pid21428.log
slave1: starting nodemanager, logging to /local/hadoop/logs/yarn-dev-nodemanager-login198.out
slave2: starting nodemanager, logging to /local/hadoop/logs/yarn-dev-nodemanager-login199.out
slave2: #
slave2: # There is insufficient memory for the Java Runtime Environment to continue.
slave2: # Native memory allocation (malloc) failed to allocate 168 bytes for AllocateHeap
slave2: # An error report file with more information is saved as:
slave2: # /local/hadoop/hs_err_pid27199.log
slave2: #
slave2: # Compiler replay data is saved as:
slave2: # /local/hadoop/replay_pid27199.log

Based on the suggestions from out of Memory Error in Hadoop and "Java Heap space Out Of Memory Error" while running a mapreduce program , I varied the heap memory size limit to 256, 512, 1024 & 2048 in all the 3 files ~/.bashrc , hadoop-env.sh and mapred-site.sh but nothing worked.

Note: I'm not an expert on Linux nor JVM.

Log file contents from one of the nodes:

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 32784 bytes for Chunk::new
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (allocation.cpp:390), pid=16375, tid=0x00007f39a352c700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_102-b14) (build 1.8.0_102-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.102-b14 mixed mode linux-amd64 compressed oops)
# Core dump written. Default location: /local/hadoop/core or core.16375 (max size 1 kB). To ensure a full core dump, try "ulimit -c unlimited" before starting Java again

CPU:total 1 (1 cores per cpu, 1 threads per core) family 6 model 45 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, clmul, tsc, tscinvbit, tscinv

Memory: 4k page, physical 2051532k(254660k free), swap 1051644k(1051324k free)

Answer 1

It's not clear from your post how much memory the VM itself has, but it seems like the VM has only 2GB of physical memory and 1GB of swap. If that be the case, you are going to really increase the memory of the VM. Absolutely nothing less than 4GB of physical memory or you'll be lucky to get the Hadoop stack running and keep the OS happy at the same time. Ideally, set each VM to about 8GB of RAM to ensure you have a few GB of RAM to throw at the MapReduce jobs.