运行本地hadoop map-reduce不会按预期对数据进行分区

Question

I have a map-reduce program that calculates the number of occurring bigrams from google ngrams on each decade.
My partitioner is:

public static class PartitionerClass extends Partitioner<Bigram, IntWritable> {
    public int getPartition(Bigram key, IntWritable value, int numPartitions) {
        String combined=key.getFirst().toString()+key.getSecond().toString()+key.getDecade().toString();
        return combined.hashCode()%numPartitions;
    }
}

I have added a breakpoint but the program does not go through that piece of code.
My main:

Configuration conf = new Configuration();
Job job = new Job(conf, "first join");
job.setJarByClass(WordCount.class);
job.setMapperClass(Map.class);
job.setReducerClass(Reduce.class);
job.setPartitionerClass(PartitionerClass.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));    ///SHOULD BE DECIDED
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapOutputKeyClass(Bigram.class);
job.setMapOutputValueClass(IntWritable.class);
System.exit(job.waitForCompletion(true) ? 0 : 1);

The code runs not as expected, some data is processed correctly and some isn't.
I really don't know how to debug this.
Any ideas?

Answer 1

The partitioner, given the number of partitions that you give, defines which key goes to which partition. Its job is not to set the number of partitions, but their contents. Each reduce tasks then handles one partition, so at the end, number of partitions = number of reduce tasks = number of output files (is using default settings and not MultipleOutputs).

In order to set the number of partitions, you should use:

job.setNumReduceTasks(n); , where n is the number that you want.

For instructions (rules of thumb, nothing strict there) on how to set this number, you can read this post .