在mapreduce作业提交期间为mapper和reducer配置内存

Question

I am trying to configure memory for mapper/reducer memory during a map reduce job submission as below:

hadoop jar Word-0.0.1-SNAPSHOT.jar -Dmapreduce.map.memory.mb=5120 com.test.Word.App /tmp/ilango/input /tmp/ilango/output/

Is there any wrong in the command above ? I am getting the following exception. It looks like do we need to put JAR file or need to configure what to use -D option in Hadoop. Thanks in advance.

Exception in thread "main" java.lang.ClassNotFoundException: -Dmapreduce.map.memory.mb=5120
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:270)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:205)

Answer 1

Command to run a MR job is

hadoop jar jarname classname input output

As per your command

hadoop jar jarname -D mapreduce.map.memory.mb=5120 classname input output

hadoop checks Driver class with name " -Dmapreduce.map.memory.mb=5120 ". Thats why it is showing java.lang.ClassNotFound Exception.

-D option should be supplied after your Driver class.

Try using below command.

hadoop jar Word-0.0.1-SNAPSHOT.jar  com.test.Word.App -D mapreduce.map.memory.mb=5120 /tmp/ilango/input /tmp/ilango/output/

Hope this solve your issue.

Answer 2

It looks like you are missing a space after -D
try -D mapreduce.map.memory.mb=5120

There is a difference between -Dproperty=value and -D property=value . The first one sets JVM system property where as the second one sets the Hadoop configuration property.

Quoting from the book Hadoop the Definitive guide, :

-D property=value Sets the given Hadoop configuration property to the given value. Overrides any default or site properties in the configuration, and any properties set via the -conf option.

Answer 3

If you're using MVN and added the Main class to the manifest, in this case com.test.Word.App , your command -D mapreduce.map.memory.mb=5120 will be taken as input.

So, just remove the com.test.Word.App line