錯誤-MapReduce中的Hadoop字數統計程序

Question

如果這看起來像是一個愚蠢的問題，我是Hadoop新手，請原諒。

我正在運行下面的MapReduce程序，並收到以下錯誤：

java.lang.Exception：java.io.IOException：映射中的鍵鍵入不匹配：預期的org.apache.hadoop.io.Text，在org.apache.hadoop.mapred.LocalJobRunner上獲得org.apache.hadoop.io.LongWritable $ Job.run（LocalJobRunner.java:354）由以下原因引起：java.io.IOException：地圖鍵中的類型不匹配：預期的org.apache.hadoop.io.Text，在組織中收到org.apache.hadoop.io.LongWritable .apache.hadoop.mapred.MapTask $ MapOutputBuffer.collect（MapTask.java:1019）

任何幫助表示贊賞。

公共類WordCount {

// Mapper Class
public static class MapperClass extends Mapper<Object, Text, Text, IntWritable>{


    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    // Mapper method defined
    public void mapperMethod(Object key,Text lineContent,Context context){
        try{
        StringTokenizer strToken = new StringTokenizer(lineContent.toString());
        //Iterating through the line 
        while(strToken.hasMoreTokens()){
            word.set(strToken.nextToken());
            try{
            context.write(word, one);
            }
            catch(Exception e){
                System.err.println(new Date()+"  ---> Cannot write data to hadoop in Mapper.");
                e.printStackTrace();
            }
        }
    }
    catch(Exception ex){
        ex.printStackTrace();
    }
    }
}
// Reducer Class
public static class ReducerClass extends Reducer<Text, IntWritable, Text, IntWritable>{

    private IntWritable result = new IntWritable();

    //Reducer method
    public void reduce(Text key,Iterable<IntWritable> values,Context context){
        try{
        int sum=0;
        for(IntWritable itr : values){
            sum+=itr.get();
        }
        result.set(sum);
        try {
            context.write(key,result);
        } catch (Exception e) {
            System.err.println(new Date()+" ---> Error while sending data to Hadoop in Reducer");
            e.printStackTrace();
        }

    }
    catch (Exception err){
        err.printStackTrace();

    }
}

}


public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
    try{
    Configuration conf = new Configuration();
    String [] arguments = new GenericOptionsParser(conf, args).getRemainingArgs();
    if(arguments.length!=2){
        System.err.println("Enter both and input and output location.");
        System.exit(1);
    }
    Job job = new Job(conf,"Simple Word Count");

    job.setJarByClass(WordCount.class);
    job.setMapperClass(MapperClass.class);
    job.setReducerClass(ReducerClass.class);


    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);



    FileInputFormat.addInputPath(job, new Path(arguments[0]));
    FileOutputFormat.setOutputPath(job, new Path(arguments[1]));

    System.exit(job.waitForCompletion(true) ? 0 : 1);
}
    catch(Exception e){

    }
}

}

Answer 1

您需要在Mapper類中重寫Map方法，而不是新方法。 由於您沒有覆蓋map方法而導致的錯誤，歸結為您的程序歸結為簡化任務。 Reducer獲取的輸入為LongWritable，Text，但是您已聲明Intwritable，且text為輸入。

希望這能解釋。

public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
         private final static IntWritable one = new IntWritable(1);
         private Text word = new Text();

         public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
       String line = value.toString();
       StringTokenizer tokenizer = new StringTokenizer(line);
           while (tokenizer.hasMoreTokens()) {
         word.set(tokenizer.nextToken());
             output.collect(word, one);
           }
         }
       }