自定义SortComparator在MapReduce wordcount程序中不起作用

Question

I am trying to understand how MapReduce Sorts the Map output keys and what is the sort algorithm which it uses. I have a text file like this

a b e f c b

how it performs the sorting with these keys. I implemented a custom SortComparator class extending WritableComparator interface. I wanted to see how sorting is happening so I am writing the operations into a file.

public static class MySortComparator2 extends WritableComparator{

  @Override
  public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {

      Text x=new Text("");
      Text y=new Text("");
      try {
        x.readFields(new DataInputStream(new ByteArrayInputStream(b1)));
        y.readFields(new DataInputStream(new ByteArrayInputStream(b2)));

        FileWriter writerOfComparisions=new FileWriter("/home/srimanth/Comparisons",true);
        writerOfComparisions.write("Comparing "+x.toString()+" with "+y.toString());
        writerOfComparisions.write(" ----> returning "+x.compareTo(y));
        writerOfComparisions.write("\n");
        writerOfComparisions.close();

    } catch (IOException e) {
        e.printStackTrace();
    }
      return x.compareTo(y);
  }

the output of the Comparisons file is this

Comparing a with a ----> returning 0
Comparing a with a ----> returning 0
Comparing a with a ----> returning 0
Comparing a with a ----> returning 0
Comparing a with a ----> returning 0
Comparing b with a ----> returning 1
Comparing c with a ----> returning 2
Comparing f with a ----> returning 5
Comparing e with a ----> returning 4
Comparing b with a ----> returning 1
Comparing b with c ----> returning -1
Comparing c with f ----> returning -3
Comparing f with e ----> returning 1
Comparing e with b ----> returning 3
Comparing b with a ----> returning 1

What is this sorting algorithm and my final output of wordcount is this

b   1
c   1
f   1
e   1
b   1
a   1

Looks like the mapreduce did not happen properly. Here is my Mapper and Reducer classes.

  public static class TokenizerMapper
       extends Mapper<Object, Text, Text, IntWritable>{

    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    public void map(Object key, Text value, Context context
                    ) throws IOException, InterruptedException {
      StringTokenizer itr = new StringTokenizer(value.toString());

      while (itr.hasMoreTokens()) {
        word.set(itr.nextToken());
        context.write(word, one);

      }
    }
  }

  public static class IntSumReducer
       extends Reducer<Text,IntWritable,Text,IntWritable> {
    private IntWritable result = new IntWritable();    
    public void reduce(Text key, Iterable<IntWritable> values,
                       Context context
                       ) throws IOException, InterruptedException {


      int sum = 0;
      for (IntWritable val : values) {
        sum += val.get();
      }

      result.set(sum);
      context.write(key, result);
    }
  }

What am I doing wrong. I would appreciate some help. Thanks in Advance.

Answer 1

Did you set the following the driver code?

job.setSortComparatorClass(MySortComparator2.class);