（Hadoop）：运行mapreduce作业时reduce方法未执行/未调用

Question

I am facing a problem in executing one my mapreduce job. As part of my map reduce task I am using mapreduce joins which includes multiple map methods and single reducer method.

Both of my map methods are getting executed, but my reducer is not getting executed/called from my driver class.

Because of this, the final output is having only the data that was collected during my map phase.

Am i using the wrong input and output values during the reduce phase? Is there any input and output mismatching between map and reduce phase?

Help me in this regard.

Here is my code..

public class CompareInputTest extends Configured implements Tool  {

public static class FirstFileInputMapperTest extends Mapper<LongWritable,Text,Text,Text>{


    private Text word = new Text();
    private String keyData,data,sourceTag = "S1$";

    public void map(LongWritable key,Text value,Context context) throws IOException, InterruptedException{

        String[] values = value.toString().split(";");
        keyData = values[1];
        data = values[2];

        context.write(new Text(keyData), new Text(data+sourceTag));


    }
}

public static class SecondFileInputMapperTest extends Mapper<LongWritable,Text,Text,Text>{
    private Text word = new Text();
    private String keyData,data,sourceTag = "S2$";
    public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{

        String[] values = value.toString().split(";");
        keyData = values[1];
        data = values[2];


        context.write(new Text(keyData), new Text(data+sourceTag));

    }

              }

public static class CounterReducerTest extends Reducer
 {
    private String status1, status2;

    public void reduce(Text key, Iterable<Text> values, Context context)
       throws IOException, InterruptedException {
        System.out.println("in reducer");

        for(Text value:values)
           {
           String splitVals[] = currValue.split("$");
        System.out.println("in reducer");
       /*
        * identifying the record source that corresponds to a commonkey and
        * parses the values accordingly
       */
      if (splitVals[0].equals("S1")) {
         status1 = splitVals[1] != null ? splitVals[1].trim(): "status1";
      } else if (splitVals[0].equals("S2")) {
          // getting the file2 and using the same to obtain the Message
          status2 = splitVals[2] != null ? splitVals[2].trim(): "status2";
      }
           }

        context.write(key, new Text(status1+"$$$"));
    }






 public static void main(String[] args) throws Exception {


     int res = ToolRunner.run(new Configuration(), new CompareInputTest(),
             args);
System.exit(res);

     }

}

public int run(String[] args) throws Exception {
    Configuration conf = new Configuration();
     Job job = new Job(conf, "count");
     job.setJarByClass(CompareInputTest.class);
     MultipleInputs.addInputPath(job,new Path(args[0]),TextInputFormat.class,FirstFileInputMapperTest.class);
     MultipleInputs.addInputPath(job,new Path(args[1]),TextInputFormat.class,SecondFileInputMapperTest.class);
     job.setReducerClass(CounterReducerTest.class);
     //job.setNumReduceTasks(1);
     job.setMapOutputKeyClass(Text.class);
     job.setMapOutputValueClass(Text.class);
     job.setOutputKeyClass(Text.class);
     job.setOutputValueClass(Text.class);




     FileOutputFormat.setOutputPath(job, new Path(args[2]));



     return (job.waitForCompletion(true) ? 0 : 1);

}

}

Answer 1

Just check the prototype of the reducer class.

extends Reducer<KEY, VALUE, KEY,VALUE>

In your case since the reducer gets as input and emits as output Text, change the definition from

public static class CounterReducerTest extends Reducer

to

public static class CounterReducerTest extends Reducer<Text,Text,Text,Text>