Hadoop MapReduce占用小文件的内存不足

Question

I'm running a MapReduce job against about 3 million small files on Hadoop (I know, I know, but there's nothing we can do about it - it's the nature of our source system).

Our code is nothing special - it uses CombineFileInputFormat to wrap a bunch of these files together, then parses the file name to add it into the contents of the file, and spits out some results. Easy peasy.

So, we have about 3 million ~7kb files in HDFS . If we run our task against a small subset of these files (one folder, maybe 10,000 files), we get no trouble. If we run it against the full list of files, we get an out of memory error.

The error comes out on STDOUT :

#
# java.lang.OutOfMemoryError: GC overhead limit exceeded
# -XX:OnOutOfMemoryError="kill -9 %p"
#   Executing /bin/sh -c "kill -9 15690"...

I'm assuming what's happening is this - whatever JVM is running the process that defines the input splits is getting totally overwhelmed trying to handle 3 million files, it's using too much memory, and YARN is killing it. I'm willing to be corrected on this theory.

So, what I need to know how to do is to increase the memory limit for YARN for the container that's calculating the input splits, not for the mappers or reducers. Then, I need to know how to make this take effect. (I've Googled pretty extensively on this, but with all the iterations of Hadoop over the years, it's hard to find a solution that works with the most recent versions...)

This is Hadoop 2.6.0, using the MapReduce API, YARN framework, on AWS Elastic MapReduce 4.2.0.

Answer 1

I would spin up a new EMR cluster and throw a larger master instance at it to see if that is the issue.

--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m3.4xlarge InstanceGroupType=CORE,InstanceCount=1,InstanceType=m3.xlarge

If the master is running out of memory when configuring the input splits you can modify the configuration EMR Configuration

Answer 2

Instead of running the MapReduce on 3 million individual files, you can merge them into manageable bigger files using any of the following approaches. 1. Create Hadoop Archive ( HAR) files from the small files. 2. Create sequence file for every 10K-20K files using MapReduce program. 3. Create a sequence file from your individual small files using forqlift tool. 4. Merge your small files into bigger files using Hadoop-Crush.

Once you have the bigger files ready, you can run the MapReduce on your whole data set.