[英]Error - Hadoop Word Count Program in MapReduce
I am new to Hadoop to so pardon me if this looks like silly question. 如果这看起来像是一个愚蠢的问题,我是Hadoop新手,请原谅。
I am running my below MapReduce program and getting the following error: 我正在运行下面的MapReduce程序,并收到以下错误:
java.lang.Exception: java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:354) Caused by: java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1019) java.lang.Exception:java.io.IOException:映射中的键键入不匹配:预期的org.apache.hadoop.io.Text,在org.apache.hadoop.mapred.LocalJobRunner上获得org.apache.hadoop.io.LongWritable $ Job.run(LocalJobRunner.java:354)由以下原因引起:java.io.IOException:地图键中的类型不匹配:预期的org.apache.hadoop.io.Text,在组织中收到org.apache.hadoop.io.LongWritable .apache.hadoop.mapred.MapTask $ MapOutputBuffer.collect(MapTask.java:1019)
Any help is appreciated. 任何帮助表示赞赏。
public class WordCount { 公共类WordCount {
// Mapper Class
public static class MapperClass extends Mapper<Object, Text, Text, IntWritable>{
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
// Mapper method defined
public void mapperMethod(Object key,Text lineContent,Context context){
try{
StringTokenizer strToken = new StringTokenizer(lineContent.toString());
//Iterating through the line
while(strToken.hasMoreTokens()){
word.set(strToken.nextToken());
try{
context.write(word, one);
}
catch(Exception e){
System.err.println(new Date()+" ---> Cannot write data to hadoop in Mapper.");
e.printStackTrace();
}
}
}
catch(Exception ex){
ex.printStackTrace();
}
}
}
// Reducer Class
public static class ReducerClass extends Reducer<Text, IntWritable, Text, IntWritable>{
private IntWritable result = new IntWritable();
//Reducer method
public void reduce(Text key,Iterable<IntWritable> values,Context context){
try{
int sum=0;
for(IntWritable itr : values){
sum+=itr.get();
}
result.set(sum);
try {
context.write(key,result);
} catch (Exception e) {
System.err.println(new Date()+" ---> Error while sending data to Hadoop in Reducer");
e.printStackTrace();
}
}
catch (Exception err){
err.printStackTrace();
}
}
}
public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
try{
Configuration conf = new Configuration();
String [] arguments = new GenericOptionsParser(conf, args).getRemainingArgs();
if(arguments.length!=2){
System.err.println("Enter both and input and output location.");
System.exit(1);
}
Job job = new Job(conf,"Simple Word Count");
job.setJarByClass(WordCount.class);
job.setMapperClass(MapperClass.class);
job.setReducerClass(ReducerClass.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(arguments[0]));
FileOutputFormat.setOutputPath(job, new Path(arguments[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
catch(Exception e){
}
}
} }
You need to override Map method in the Mapper Class instead you have a new method. 您需要在Mapper类中重写Map方法,而不是新方法。 Comming to your error, as you dont have map method overridden your program boils down to a reduce only job.
由于您没有覆盖map方法而导致的错误,归结为您的程序归结为简化任务。 Reducer is getting input as LongWritable,Text but you have declared Intwritable and text as input.
Reducer获取的输入为LongWritable,Text,但是您已声明Intwritable,且text为输入。
Hope this explains. 希望这能解释。
public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
output.collect(word, one);
}
}
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.