在Hadoop-2.6.0中运行我自己的WordCount.java版本

Question

I am trying to create my own version of wordcount and execute it. For that, I am trying to create the wordcount.jar by executing the following command (as described here http://cs.smith.edu/dftwiki/index.php/Hadoop_Tutorial_1_--_Running_WordCount for previous releases than Hadoop-2.*):

javac -classpath /usr/local/hadoop-2.6.0/share/hadoop/common/*:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/* -d wordcount_classes/ WordCount.java
jar -cvf wordcount.jar -C wordcount_classes/ .

The problem is that I get errors in the first command when trying to compile the wordcount class. In the first command I try to include all the jars that exist in the common and mapreduce directories, because I don't find any jar that has the name "hadoop-0.19.2-core.jar" as described in tutorials I found in the internet for the hadoop-0.*

 javac -classpath /home/hadoop/hadoop/hadoop-0.19.2-core.jar -d wordcount_classes WordCount.java 

When I execute the given wordcount as follows, it works perfect, but I can't compile my own version:

 hadoop jar /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-*-examples-2.6.0.jar wordcount input output

I've searched in the internet but I find most of tutorials are for Hadoop-1.* which is different from the 2.6.0 version which does not contain any java source file such as the wordcount.java I need, and also has a completely different structure. Hence, I was obliged to download the wordcount.java file separately from https://github.com/apache/hadoop-common/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/WordCount.java and copy the WordCount.java in the current directory where I execute my commands to create the jar.

I've downloaded Hadoop from this source http://mirror.its.dal.ca/apache/hadoop/common/hadoop-2.6.0/

Here is the details of the errors I get:

  /usr/local/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0.jar(org  /apache/hadoop/fs/Path.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate': class file for org.apache.hadoop.classification.InterfaceAudience not found
  WordCount.java:54: error: cannot access Options
String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
                     ^
class file for org.apache.commons.cli.Options not found
WordCount.java:68: error: method waitForCompletion in class Job cannot be applied to given types;
System.exit(job.waitForCompletion() ? 0 : 1);
               ^
required: boolean
found: no arguments
reason: actual and formal argument lists differ in length
Note: WordCount.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
2 errors
1 warning

Here is also the WordCount.java :

//package org.apache.hadoop.examples;
package org.myorg;

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class WordCount {

public static class TokenizerMapper 
   extends Mapper<Object, Text, Text, IntWritable>{

private final static IntWritable one = new IntWritable(1);
private Text word = new Text();

public void map(Object key, Text value, Context context
                ) throws IOException, InterruptedException {
  StringTokenizer itr = new StringTokenizer(value.toString());
  while (itr.hasMoreTokens()) {
    word.set(itr.nextToken());
    context.write(word, one);
   }
 }
}

public static class IntSumReducer 
   extends Reducer<Text,IntWritable,Text,IntWritable> {
private IntWritable result = new IntWritable();

public void reduce(Text key, Iterable<IntWritable> values, 
                   Context context
                   ) throws IOException, InterruptedException {
  int sum = 0;
  for (IntWritable val : values) {
    sum += val.get();
  }
  result.set(sum);
  context.write(key, result);
  }
}

public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
if (otherArgs.length != 2) {
  System.err.println("Usage: wordcount <in> <out>");
  System.exit(2);
}
Job job = new Job(conf, "word count");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
System.exit(job.waitForCompletion() ? 0 : 1);
 }
}

Thank you in advance for your help

Answer 1

This looks more like a Java problem than a Hadoop problem to me. I would recommend to use Eclipse and Hadoop Developer Tools that works with Hadoop version 2.2.0. (unless you specifally need 2.6.0).

Also you may try the %HADOOP_HOME%\\share\\hadoop\\mapreduce\\hadoop-mapreduce-client-core- version_number .jar

Also the first error message you get at line 54 may be resolved by implementing org.apache.hadoop.util.Tool Class info for Tool I found this info at Apache docs on options .

At line 68 it you're missing an argument for the job.waitForCompletion() method call. I would suggest to pass a true argument to this method.

Answer 2

I resolved the problem by doing the following (It is even easier now with Hadoop-2.6.0) (see this tutorial for more details http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Usage ):

• Add the following to your ~/.profile file (don't forget to execute "source ~/.profile" afterwards):

  export JAVA_HOME=/usr/java/default

  export PATH=$JAVA_HOME/bin:$PATH

  export HADOOP_CLASSPATH=$JAVA_HOME/lib/tools.jar # This is very important to facilitate compilation of your java classes.

• I then compile WordCount.java and create a jar:

  $ bin/hadoop com.sun.tools.javac.Main WordCount.java

  $ jar cf wc.jar WordCount*.class

• Then just use a command like this to execute your job:

  $ hadoop jar wc.jar WordCount INPUT/ OUTPUT/

Answer 3

Try this:

javac --classpath `hadoop classpath` .... (rest of command)

Test on ubuntu and debian.

This work fine for me.

Answer 4

there is small correction at System.exit(job.waitForCompletion() ? 0 : 1); it should be like this System.exit(job.waitForCompletion(true) ? 0 : 1); because it will return boolean value..

Answer 5

我解决了将位于以下位置的jar添加到WordCount项目中的问题：share / hadoop / common / lib / commons-cli-1.2.jar