Hi I am new to Hadoop MR. I tried to write a simple MR job to count the shortest path for a node to its destination node.Basically the logic is like :
If the input text file has the following paths given: ABCD ABD ACD BED BD BACD
The output should be as : ABD BD
which just gives the shortest path between nodes A and D and the shortest path between B and D.
The output I am getting is : [ABCD ABD ACD BED BD BACD]
I have wrote the following MR to do the same. But its not giving the desired answer. I am running the MR in stand alone mode.
Please let me know what is wrong with the code and the solution for it. Thanks a lot for your time.
public class Shpath {
public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, Text> {
public void map(LongWritable key, Text value, OutputCollector<Text, Text> output, Reporter reporter) throws IOException {
String[] line = value.toString().split("\t");
List<String> l = new ArrayList<String>();
for(String lin :line){
l.add(lin);
}
List <String>startEnd = new ArrayList<String>();
for(String s : l){
String g = s.substring(0,1)+s.substring((s.length())-1);
if(!startEnd.contains(g))
{
startEnd.add(g);
}
}
List <String> uniqueStringList = new ArrayList<String>();
java.util.Map finalMap = new HashMap();
for(String s1 : startEnd){
for(String s : l) {
if(s.startsWith(s1.substring(0,1)) && (s.endsWith(s1.substring((s1.length())-1)))){
uniqueStringList.add(s);
}
}
String smallestKey = null;
int minSize = Integer.MAX_VALUE;
String smallest = null;
for(String s2 : uniqueStringList){
if(s2.length() < minSize) {
minSize = s2.length();
smallest = s2;
smallestKey = s1;
}
finalMap.put(s1,smallest);
}
uniqueStringList.clear();
}output.collect(new Text(),new Text(finalMap.values().toString()));
}
}
public static class Reduce extends MapReduceBase implements Reducer<Text, Text, Text, Text> {
public void reduce(Text key, Iterator<Text> value, OutputCollector<Text, Text> output, Reporter reporter) throws IOException {
while (value.hasNext()){
output.collect(new Text(key),new Text(value.next()));
}
}
}
public static void main(String[] args) throws Exception {
JobConf conf = new JobConf(Shpath.class);
conf.setJobName("shpath");
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(Text.class);
conf.setMapperClass(Map.class);
conf.setCombinerClass(Reduce.class);
conf.setReducerClass(Reduce.class);
conf.setInputFormat(org.apache.hadoop.mapred.TextInputFormat.class);
conf.setOutputFormat(org.apache.hadoop.mapred.TextOutputFormat.class);
org.apache.hadoop.mapred.FileInputFormat.setInputPaths(conf, new Path(args[0]));
org.apache.hadoop.mapred.FileOutputFormat.setOutputPath(conf, new Path(args[1]));
JobClient.runJob(conf);
}
}
I am not sure but it has to be something like this:
public void map(LongWritable key, Text value, OutputCollector<Text, Text> output, Reporter reporter) throws IOException {
Map<String , HashMap<Integer, String> > outMap = new HashMap<String, HashMap<Integer, String> >();
HashMap<Integer, String> tempMap = new HashMap<Integer, String>();
tempMap.put(Integer.MAX_VALUE, "");
outMap.put("AD", tempMap);
outMap.put("BD", tempMap);
String[] line = value.toString().split("\t");
for (String path : line) {
String tempPath = new String( new char[]{path.charAt(0) , path.charAt(path.length() - 1)});
if(outMap.containsKey(tempPath)) {
HashMap<Integer, String> tempOutMap = outMap.get(tempPath);
for (Iterator itr = tempOutMap.keySet().iterator(); itr.hasNext(); ) {
Integer count = (Integer) itr.next();
if(count > tempPath.length()){
tempMap.remove(count);
tempMap.put(tempPath.length(), tempPath);
}
}
}
}
for (String str : outMap.keySet()) {
output.collect(new Text(str), new Text(outMap.get(str).values().toString()));
}
}
public void reduce(Text key, Iterator<Text> value, OutputCollector<Text, Text> output, Reporter reporter) throws IOException {
String outString;
int smallest = Integer.MAX_VALUE;
while (value.hasNext()){
String str = value.next();
if(str.length() < smallest) {
outString = str;
smallest = str.length();
}
}
output.collect(new Text(key),new Text(outString));
}
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.