运行hadoop jar时发生ClassNotFoundException

Question

I'm attempting to run a MapReduce job from a jar file and keep getting a ClassNotFoundException error. I'm running Hadoop 1.2.1 on a Centos 6 virtual machine.

First I compiled the file exercise.java (and class) into a jar file exercise.jar using the following shell script compile.sh :

#!/bin/bash

javac -classpath /pathto/hadoop-common-1.2.1.jar:\
/pathto/hadoop-core-1.2.1.jar /pathto/exercise.java

jar cvf exercise.jar /pathto/*.class

This runs fine and the jar completes successfully. I then attempt to run the actual MapReduce job using shell script exec.sh :

#!/bin/bash

export CLASSPATH=$CLASSPATH:/pathto/hadoop-common-1.2.1.jar:\
/pathto/hadoop-core-1.2.1.jar:/pathto/exercise.class

hadoop jar exercise.jar exercise /data/input/inputfile.txt /data/output

This trows the ClassNotFoundException error :

Exception in thread "main" java.lang.ClassNotFoundException: exercise
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:274)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:153) 

I realize the explicit path names might not be necessary but I've been a little desperate to double check everything. I've confirmed that in my exercise.java file exercise.class is in the job configuration via job.setJarByClass(exercise.class); and confirmed exercise.class is contained in exercise.jar . Can't seem to figure it out.

UPDATE The exec.sh script with the full path of exercise.class . It's stored in my Eclipse project directory:

#!/bin/bash

export CLASSPATH=$CLASSPATH:/pathto/hadoop-common-1.2.1.jar:\
/pathto/hadoop-core-1.2.1.jar:/home/username/workspace/MVN_Hadoop/src/main/java.com.amend.hadoop.MapReduce/*

hadoop jar \
exercise.jar \
/home/username/workspace/MVN_Hadoop/src/main/java.com.amend.hadoop.MapReduce/exercise \
/data/input/inputfile.txt \
/data/output

When I actually try and run the exec.sh script using the explicitly written out path names, I also get a completely different set of errors:

Exception in thread "main" java.lang.ClassNotFoundException: /home/hdadmin/workspace/MVN_Hadoop/src/main/java/come/amend/hadoop/MapReduce/exercise
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:274)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:153) 

Answer 1

I could see this possible errors. From the Hadoop jar exercise.jar exercise /data/input/inputfile.txt /data/output please specify the full path of the exercise class. Ie org.name.package.exercise if exists. To cross check open the jar file and check the location of exercise.class location. To continue, Hadoop doesn't expect jars to be included within the jars, since the path of Hadoop is set globally.

NEW: See, the following path is some thing weird. "/home/hdadmin/workspace/MVN_Hadoop/src/main/java/come/amend/hadoop/MapReduce/exercise"

If you are running using your jar, how could a class path be so specific, instead of jar path. It could only be "come/amend/hadoop/MapReduce/exercise" this.