Compile Hadoop 2.2.0 job?

Question

It seems that all of the examples are constructed with older versions in mind.

How do I compile my java program on Ubuntu such that it will refer to hadoop-2.2.0 libraries?

Where are the jar files that I am supposed to include?

What is the command? Is it like -

javac -classpath libraries wordcount.java

Thank you.

Answer 1

The simplest solution for Linux machines would be:

javac -classpath `yarn classpath` -d . WordCount.java

Or:

export CLASSPATH=`yarn classpath`
javac -classpath $CLASSPATH -d . WordCount.java

Answer 2

I found the following:

javac -classpath $HADOOP_HOME/share/hadoop/common/hadoop-common-2.2.0.jar:$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.2.0.jar:$HADOOP_HOME/share/hadoop/common/lib/commons-cli-1.2.jar -d wordcount_classes myWordCount.java

This allowed me to compile the Wordcount example (or in this case a copy of mine called myWordCount).

Answer 3

Hadoop has a command "hadoop classpath" that supplies you with the necessary classpath. ie

hadoop classpath /etc/hadoop/conf:/usr/lib/hadoop/lib/ :/usr/lib/hadoop/.// :/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/ :/usr/lib/hadoop-hdfs/.// :/usr/lib/hadoop-yarn/lib/ :/usr/lib/hadoop-yarn/.// :/usr/lib/hadoop-mapreduce/lib/ :/usr/lib/hadoop-mapreduce/.//

So if you wanna compile you can use it this way..

javac -classpath $(hadoop classpath) -d . WordCount.java

Answer 4

你必须安装Cygin，你可以运行你的hadoop示例，你也可以用eclipse配置你的hadoop

Answer 5

Run the command: "yarn classpath" to see a list of directories. When I use this list as my -classpath option for javac, my Java program compiles.

I am running HortonWorks v2.0, Apache Hadoop 2.2.0.

Answer 6

I'm having bumpy ride with Hadoop Example jars too. Information in many videos/tutorials/blogs is based on older version.

When we compile these examples or write any of our own MapReduce program, that is going to use hadoop packages (ie import jar in IDE/add reference to external jars - akin to Add reference to .dll in MS Visual Studio), and IDE will take care of correctly calling javac for each class.

Now for manually compiling any class eg WordCount.java, we need to tell javac which all jars our class is dependent on. I followed outdated videos but that shared one information ie to set a variable in .bashrc, having reference to all Hadoop related jar files and then use that in javac -classpath $VARIABLE filename.java.

eg I'm using name as $HADOOP_CLASSPATH and values as shown here (I'm on Mac OS X)

/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/ :/usr/local/hadoop/share/hadoop/common/ :/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/ :/usr/local/hadoop/share/hadoop/hdfs/ :/usr/local/hadoop/share/hadoop/yarn/lib/ :/usr/local/hadoop/share/hadoop/yarn/ :/usr/local/hadoop/share/hadoop/mapreduce/lib/ :/usr/local/hadoop/share/hadoop/mapreduce/ :/contrib/capacity-scheduler/ .jar:/usr/local/hadoop/share/hadoop/yarn/ :/usr/local/hadoop/share/hadoop/yarn/lib/*

with this variable, I could compile class successfully. "javac -classpath $HADOOP_CLASSPATH WordCount.java "