WordCount MapReduce提供了意外结果

Question

I am trying this java code for wordcount in mapreduce and after completion of reduce method I want to display the only word that comes maximum number of times.

For that i have created some class level variables named as myoutput, mykey and completeSum.

I am writing this data in close method but I am getting unexpected result at the end.

public class WordCount {

public static class Map extends MapReduceBase implements
        Mapper<LongWritable, Text, Text, IntWritable> {
    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    public void map(LongWritable key, Text value,
            OutputCollector<Text, IntWritable> output, Reporter reporter)
            throws IOException {
        String line = value.toString();
        StringTokenizer tokenizer = new StringTokenizer(line);

        while (tokenizer.hasMoreTokens()) {
            word.set(tokenizer.nextToken());
            output.collect(word, one);
        }

    }
}

static int completeSum = -1;
static OutputCollector<Text, IntWritable> myoutput;
static Text mykey = new Text();

public static class Reduce extends MapReduceBase implements
        Reducer<Text, IntWritable, Text, IntWritable> {

    public void reduce(Text key, Iterator<IntWritable> values,
            OutputCollector<Text, IntWritable> output, Reporter reporter)
            throws IOException {
        int sum = 0;
        while (values.hasNext()) {
            sum += values.next().get();
        }

        if (completeSum < sum) {
            completeSum = sum;
            myoutput = output;
            mykey = key;
        }


    }

    @Override
    public void close() throws IOException {
        // TODO Auto-generated method stub
        super.close();
        myoutput.collect(mykey, new IntWritable(completeSum));
    }
}

public static void main(String[] args) throws Exception {

    JobConf conf = new JobConf(WordCount.class);
    conf.setJobName("wordcount");

    conf.setOutputKeyClass(Text.class);
    conf.setOutputValueClass(IntWritable.class);

    conf.setMapperClass(Map.class);
    // conf.setCombinerClass(Reduce.class);
    conf.setReducerClass(Reduce.class);

    conf.setInputFormat(TextInputFormat.class);
    conf.setOutputFormat(TextOutputFormat.class);

    FileInputFormat.setInputPaths(conf, new Path(args[0]));
    FileOutputFormat.setOutputPath(conf, new Path(args[1]));

    JobClient.runJob(conf);

}
}

---

input file data

one 
three three three
four four four four 
 six six six six six six six six six six six six six six six six six six 
five five five five five 
seven seven seven seven seven seven seven seven seven seven seven seven seven 

---

result should come as

six 18

---

however I am getting this result

three 18

---

By the result I can see that the sum is correct but the key is not.

If someone can give good reference on these map and reduce methods, that would be very helpful.

Answer 1

The problem you are observing is due to reference aliasing. The object referenced by the key is reused with a new content for multiple invocations, thus changing mykey that references the same object. It ends up with the last reduced key. This could be avoided by copying the object, as in:

mykey = new Text(key);

However, you should get the result only from the output file as static variables cannot be shared by different nodes in a distributed cluster. It sort of works only in standalone mode, defeating the purpose of map-reduce.

Finally, using global variables, even in standalone mode, will most likley lead to races if using parallel local tasks (see MAPREDUCE-1367 and MAPREDUCE-434 ).