classpath issue in hadoop java

Question

javac -classpath "/installs/hadoop-0.20.2/*.jar;/installs/hadoop-0.20.2/lib/*.jar" appClientModule/grid/comp/tools/CleanTmp.java
appClientModule/grid/comp/tools/CleanTmp.java:2: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.*;
^
appClientModule/grid/comp/tools/CleanTmp.java:3: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.JobConf;
                               ^
appClientModule/grid/comp/tools/CleanTmp.java:4: package org.apache.commons.logging does not exist
import org.apache.commons.logging.*;
^
appClientModule/grid/comp/tools/CleanTmp.java:5: package org.apache.commons.logging.impl does not exist
import org.apache.commons.logging.impl.Log4JLogger;
                                      ^
appClientModule/grid/comp/tools/CleanTmp.java:8: cannot find symbol
symbol  : class Log4JLogger
location: class grid.comp.tools.CleanTmp
    Log4JLogger lg=new Log4JLogger(this.getClass().getName());
    ^
appClientModule/grid/comp/tools/CleanTmp.java:9: cannot find symbol
symbol  : class JobConf
location: class grid.comp.tools.CleanTmp
    JobConf jconf;
    ^
appClientModule/grid/comp/tools/CleanTmp.java:8: cannot find symbol
symbol  : class Log4JLogger
location: class grid.comp.tools.CleanTmp
    Log4JLogger lg=new Log4JLogger(this.getClass().getName());
                       ^
7 errors
tools $ javac -classpath "/installs/hadoop-0.20.2/*.jar;/installs/hadoop-0.20.2/lib/*.jar" appClientModule/grid/comp/tools/CleanTmp.java
jabir:CompareHdfs jabir.ahmed$ javac -classpath "/installs/hadoop-0.20.2/*.jar;/installs/hadoop-0.20.2/lib/*.jar" appClientModule/grid/comp/tools/CleanTmp.java
appClientModule/grid/comp/tools/CleanTmp.java:2: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.*;
^
appClientModule/grid/comp/tools/CleanTmp.java:3: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.JobConf;
                               ^
appClientModule/grid/comp/tools/CleanTmp.java:4: package org.apache.commons.logging does not exist
import org.apache.commons.logging.*;
^
appClientModule/grid/comp/tools/CleanTmp.java:5: package org.apache.commons.logging.impl does not exist
import org.apache.commons.logging.impl.Log4JLogger;
                                      ^
appClientModule/grid/comp/tools/CleanTmp.java:8: cannot find symbol
symbol  : class Log4JLogger
location: class grid.comp.tools.CleanTmp
    Log4JLogger lg=new Log4JLogger(this.getClass().getName());
    ^
appClientModule/grid/comp/tools/CleanTmp.java:9: cannot find symbol
symbol  : class JobConf
location: class grid.comp.tools.CleanTmp
    JobConf jconf;
    ^
appClientModule/grid/comp/tools/CleanTmp.java:8: cannot find symbol
symbol  : class Log4JLogger
location: class grid.comp.tools.CleanTmp
    Log4JLogger lg=new Log4JLogger(this.getClass().getName());
                       ^
7 errors
tools $ ls /installs/hadoop-0.20.2/lib/*.jar
/installs/hadoop-0.20.2/lib/ant-contrib-1.0b3.jar           /installs/hadoop-0.20.2/lib/jasper-runtime-5.5.12.jar
/installs/hadoop-0.20.2/lib/aspectjrt-1.6.5.jar             /installs/hadoop-0.20.2/lib/jets3t-0.6.1.jar
/installs/hadoop-0.20.2/lib/aspectjtools-1.6.5.jar          /installs/hadoop-0.20.2/lib/jetty-6.1.26.cloudera.1.jar
/installs/hadoop-0.20.2/lib/commons-cli-1.2.jar             /installs/hadoop-0.20.2/lib/jetty-servlet-tester-6.1.26.cloudera.1.jar
/installs/hadoop-0.20.2/lib/commons-codec-1.4.jar           /installs/hadoop-0.20.2/lib/jetty-util-6.1.26.cloudera.1.jar
/installs/hadoop-0.20.2/lib/commons-daemon-1.0.1.jar            /installs/hadoop-0.20.2/lib/jsch-0.1.42.jar
/installs/hadoop-0.20.2/lib/commons-el-1.0.jar              /installs/hadoop-0.20.2/lib/junit-4.5.jar
/installs/hadoop-0.20.2/lib/commons-httpclient-3.1.jar          /installs/hadoop-0.20.2/lib/kfs-0.2.2.jar
/installs/hadoop-0.20.2/lib/commons-logging-1.0.4.jar           /installs/hadoop-0.20.2/lib/log4j-1.2.15.jar
/installs/hadoop-0.20.2/lib/commons-logging-api-1.0.4.jar       /installs/hadoop-0.20.2/lib/mockito-all-1.8.2.jar
/installs/hadoop-0.20.2/lib/commons-net-1.4.1.jar           /installs/hadoop-0.20.2/lib/oro-2.0.8.jar
/installs/hadoop-0.20.2/lib/core-3.1.1.jar              /installs/hadoop-0.20.2/lib/servlet-api-2.5-20081211.jar
/installs/hadoop-0.20.2/lib/hadoop-fairscheduler-0.20.2-cdh3u2.jar  /installs/hadoop-0.20.2/lib/servlet-api-2.5-6.1.14.jar
/installs/hadoop-0.20.2/lib/hsqldb-1.8.0.10.jar             /installs/hadoop-0.20.2/lib/slf4j-api-1.4.3.jar
/installs/hadoop-0.20.2/lib/jackson-core-asl-1.5.2.jar          /installs/hadoop-0.20.2/lib/slf4j-log4j12-1.4.3.jar
/installs/hadoop-0.20.2/lib/jackson-mapper-asl-1.5.2.jar        /installs/hadoop-0.20.2/lib/xmlenc-0.52.jar
/installs/hadoop-0.20.2/lib/jasper-compiler-5.5.12.jar## Heading ##

So how do i set the class path

I tried setting via the env variable too

$ echo $CLASSPATH /installs/hadoop/lib/ .jar:/installs/hadoop/ .jar:/installs/hadoop-0.20.2/ .jar:/installs/hadoop-0.20.2/lib/ .jar

it still fails

$ javac -version javac 1.6.0_29

Answer 1

Go for a more clear classpath like:

javac -classpath $HADOOP_HOME/hadoop-core-0.20.204.0.jar:$HADOOP_HOME/lib/commons-cli-1.2.jar -d inception src/Inception.java

(in this case Inception is your java program. don't know why i mentioned that but it's better to be more explicit then not!)

Most of the time you are going to need the hadoop-core-(yourversion) and the commons-cli if you are using the Tool interface.

Answer 2

inquire's solution helped me. Here is the syntax I used on HDInsight version 2.1:

C:\\apps\\dist\\java\\bin\\javac -classpath %HADOOP_HOME%\\hadoop-core-1.2.0.1.3.0.1-0302.jar;%HADOOP_HOME%\\lib\\commons-cli-1.2.jar WordCount.java

Answer 3

Instead of specifying individual libraries, just add " -classpath yarn classpath ". That makes the command pretty easy and simple.

Answer 4

If you have set the CLASSPATH variable, then you don't need to specify the -classpath flag in the javac command as that will take the classpath value from the environment variable CLASSPATH

The value of classpath needs to point to the actual jar file rather than .jar For example,

set CLASSPATH=/myapp/mylib.jar

Depending on the platform you are on, you can either use : or ; as the delimiter between different paths. For Windows, use ; and UNIX normally uses :

In UNIX:

set CLASSPATH=/myapp/myclasses1.jar:/myapp/myclasses2.jar

But in Windows you would need to use semicolon as the delimiter between the paths

set CLASSPATH=/myapp/myclasses1.jar;/myapp/myclasses2.jar