I am trying to implement in memory sorting,ie soring based on the no. of counts. I am facing the below problem, null pinter exception in output.collect in the close method of the Reduce class. Pls help!
Is my coding logic correct? I am keeping in memory the tokens from different instances of reduce method. Please help me! I wanted a sorted output of based on the TCounts.
package com.a;
import java.io.IOException;
import java.util.ArrayList;
import java.util.StringTokenizer;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;
public class Map1 extends MapReduceBase implements Mapper<LongWritable, Text, Text,
Text >{
public void map(LongWritable key, Text value, OutputCollector<Text, Text> output,
Reporter reporter) throws IOException {
StringTokenizer tokenizer = new StringTokenizer(value.toString());
String tk = tokenizer.nextToken();
String id = tokenizer.nextToken();
String name = tokenizer.nextToken();
StringTokenizer tkz = new StringTokenizer(name, ",");
ArrayList<String> al = new ArrayList<String>();
while(tkz.hasMoreTokens())
{
name = tkz.nextToken();
al.add(name);
}
for(int i = 0; i<al.size(); i++)
{
output.collect(new Text(t+" "+al.get(i)), new Text("1"));
System.out.println("out key:----->"+t+" "+al.get(i));
}
}
}
public class Reduce1 extends MapReduceBase implements Reducer<Text, Text, Text, Text>{
// @SuppressWarnings("unchecked")
ArrayList<TCount> al = new ArrayList<TCount>();
String key_str = null;
private OutputCollector<Text, Text> output;
public void reduce (Text key, Iterator<Text> values, OutputCollector<Text,
Text> output, Reporter reporter) throws IOException {
int sum = 0;
while(values.hasNext())
{
String val = values.next().toString();
sum = sum+Integer.parseInt(val);;
}
String str_val = String.valueOf(sum);
key_str = key.toString();
//output.collect(key, new Text(str_val));
TCount tc = new TCount(key.toString(), sum);
al.add(tc);
}
private Text t = new Text();
private Text txt_key = new Text();
public void close() throws IOException {
Collections.sort(al);
for(int i = 0; i<al.size(); i++)
{
String tkn = al.get(i).getT();
System.out.println("token:-------------------> "+tkn);
System.out.println("output: "+output);
txt_key = new Text(t);
txt = new Text(String.valueOf(al.get(i).getCount()));
output.collect(txt_key, t);
}
}
}
In class Reduce1, you're declaring the output object:
private OutputCollector<Text, Text> output;
without initializing it -> so it's null at this time.
Also in the method reduce()
you pass a parameter of the same type ( OutputCollector<Text, Text> output
) -> so in this method, I guess you wanna say:
this.output=output;
// if the object is null, initialize it if you wanna use it
The thing is you'll get a null pointer exception as long as your object is not initialized (instantiated).
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.