尝试将整个文件夹作为Mapreduce程序的输入时，我遇到“ Java堆空间错误”

Question

I'm facing "Java Heap space error",when I'm trying to run the mapreduce program by giving entire folder as input to the MR Job.When I'm giving a single file as input to MR job,I'm facing no error.The job has run successfully.

Changes I tried in hadoop-env.sh file:
=====================================
I had increased the memory size from 1024 to 2048MB
export HADOOP_CLIENT_OPTS="-Xmx2048m $HADOOP_CLIENT_OPTS"

Changes in mapred-site.xml:
===========================
<property>
  <name>mapred.child.java.opts</name>
  <value>-Xmx2048m</value>
</property>

By making changes in these files also,I'm still facing the "Java heap space error".

Can anyone please suggest me on this issue ...

Answer 1

You can turn on the HPROF profiling for your job with something like this,

conf.setBoolean("mapred.task.profile", true); conf.set("mapred.task.profile.params", "-agentlib:hprof=cpu=samples," + "heap=sites,depth=6,force=n,thread=y,verbose=n,file=%s"); conf.set("mapred.task.profile.maps", "0-2"); conf.set("mapred.task.profile.reduces", "0-2");

This will help you to diagnose what exhausted the heap. See more details in "Hadoop The Definitive Guide" page 178-181."