I am trying to configure memory for mapper/reducer memory during a map reduce job submission as below:
hadoop jar Word-0.0.1-SNAPSHOT.jar -Dmapreduce.map.memory.mb=5120 com.test.Word.App /tmp/ilango/input /tmp/ilango/output/
Is there any wrong in the command above ? I am getting the following exception. It looks like do we need to put JAR file or need to configure what to use -D option in Hadoop. Thanks in advance.
Exception in thread "main" java.lang.ClassNotFoundException: -Dmapreduce.map.memory.mb=5120
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.hadoop.util.RunJar.main(RunJar.java:205)
Command to run a MR job is
hadoop jar jarname classname input output
As per your command
hadoop jar jarname -D mapreduce.map.memory.mb=5120 classname input output
hadoop checks Driver class with name " -Dmapreduce.map.memory.mb=5120 ". Thats why it is showing java.lang.ClassNotFound Exception.
-D option should be supplied after your Driver class.
Try using below command.
hadoop jar Word-0.0.1-SNAPSHOT.jar com.test.Word.App -D mapreduce.map.memory.mb=5120 /tmp/ilango/input /tmp/ilango/output/
Hope this solve your issue.
It looks like you are missing a space after -D
try -D mapreduce.map.memory.mb=5120
There is a difference between -Dproperty=value and -D property=value . The first one sets JVM system property where as the second one sets the Hadoop configuration property.
Quoting from the book Hadoop the Definitive guide, :
-D property=value Sets the given Hadoop configuration property to the given value. Overrides any default or site properties in the configuration, and any properties set via the -conf option.
If you're using MVN and added the Main class to the manifest, in this case com.test.Word.App
, your command -D mapreduce.map.memory.mb=5120
will be taken as input.
So, just remove the com.test.Word.App
line
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.