I want to do a total sort in mapreduce Word Count.
public int run(String[] args) throws Exception {
Job job = Job.getInstance(getConf(), "wordcount");
job.setJarByClass(this.getClass());
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setMapperClass(Map.class);
//Total Sort
job.setPartitionerClass(TotalOrderPartitioner.class);
InputSampler.Sampler<Text, IntWritable> sampler = new InputSampler.RandomSampler<Text, IntWritable>(0.1, 10000, 10);
InputSampler.writePartitionFile(job, sampler);
Path inputDir = new Path(args[2] + "/_tmp");
Path partitionFile = new Path(inputDir, "_partitioning");
TotalOrderPartitioner.setPartitionFile(job.getConfiguration(),partitionFile);
InputSampler.writePartitionFile(job, sampler);
job.setReducerClass(Reduce.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
return job.waitForCompletion(true) ? 0 : 1;
}
But i got error like java.io.IOException: wrong key class: org.apache.hadoop.io.Text is not class org.apache.hadoop.io.LongWritable
I don't understand how InputSampler.RandomSampler work.
In the above code there is no InputFormat
set for the job, so default will be taken which is TextInputFormat<LongWritable,Text>
.
For InputSampler.RandomSampler<Text, IntWritable>
it has been configured as Text, IntWritable which doesn't match with TextInputFormat
.
Since there is type mismatch between InputFormat
and InputSampler
the error is thrown.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.