In a Map-Reduce job, I am using five different files where in my dataset contains values under two categories P
and I
. After I
specific values are found, I am passing those into I-part-r-00000 file and accordingly, for P. I am using MultipleOutputformat class in reducer to achieve this.
My Mapper class contains:
public class parserMapper extends Mapper<LongWritable, Text, Text, Text> {
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String IPFLAG = "";
String[] element_data= value.toString.split(",");
if(element_data[0].toString().trim().equalsIgnoreCase("005010X222A1")){
IPFLAG = "P";
}
else {
IPFLAG = "I";
}
if (IPFLAG == "P") {
context.write(new Text(IPFLAG), new Text(theData));
}
else if (IPFLAG == "I") {
context.write(new Text(IPFLAG), new Text(theData));
}
else{
System.out.println("No category found");
}
}
public void run(Context context) throws IOException, InterruptedException {
setup(context);
while (context.nextKeyValue()) {
map(context.getCurrentKey(), context.getCurrentValue(), context);
}
cleanup(context);
}
}
Reducer class includes:
public class parserReducer extends Reducer<Text, Text, Text, Text> {
private MultipleOutputs multipleOutputs;
@Override
protected void setup(Context context) throws IOException, InterruptedException {
multipleOutputs = new MultipleOutputs(context);
}
@Override
protected void cleanup(Context context) throws IOException, InterruptedException {
multipleOutputs.close();
}
@Override
public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
Object c = null;
try{
if (!(key.toString().isEmpty())) {
for (Text value : values) {
multipleOutputs.write(c, value, key.toString());
}
}
}
catch(Exception e){ System.out.println("Caught Exception: " + e.getMessage());}
}
}
and Driver code includes =>
public class parserDriver {
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
conf.set("textinputformat.record.delimiter", "~"+"\n"+"ISA*");
Job job = new Job(conf);
job.setJobName("PARSER");
job.setJarByClass(parserDriver.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
job.setMapperClass(parserMapper.class);
job.setReducerClass(parserReducer.class);
// job.setOutputFormatClass(TextOutputFormat.class);
LazyOutputFormat.setOutputFormatClass(job, TextOutputFormat.class);
// job.setOutputFormatClass(LazyOutputFormat.class);
/* MultipleOutputs.addNamedOutput(job, "P", TextOutputFormat.class, Text.class, Text.class);
MultipleOutputs.addNamedOutput(job, "I", TextOutputFormat.class, Text.class, Text.class);
*/
// Pass as option -D mapred.reduce.tasks=<number>
job.setNumReduceTasks(3);
/* This line is to accept the input recursively */
//FileInputFormat.setInputDirRecursive(job, true);
FileInputFormat.addInputPath(job, "/Users/Mohit/input");
FileOutputFormat.setOutputPath(job, "/Users/Mohit/output");
/*
* Delete output file path if already exists
*/
FileSystem fs = FileSystem.get(conf);
if (fs.exists(outputFilePath)) {
fs.delete(outputFilePath, true);
}
return job.waitForCompletion(true) ? 0: 1;
}
}
Through all this, I am trying to achieve two partitions against a single file
file1 -> P-part-r00000, I-part-r00001
file2 -> P-part-r00002, I-part-r00003
. but I am getting two partitions against all the files being fed as input to this job.
file1, file2, file3, file4, file5 -> P-part-r00000, I-part-r00001
Not sure what am I missing here, if anybody can help please?
1) In your Driver add these lines to file naming:
job.setOutputFormatClass(TextOutputFormat.class);
MultipleOutputs.addNamedOutput(job, "I", TextOutputFormat.class,
Text.class, Text.class);
MultipleOutputs.addNamedOutput(job, "P", TextOutputFormat.class,
Text.class, Text.class);
2) Change your reducer to send each value to file with specific name:
@Override
public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
try{
if (!(key.toString().isEmpty())) {
for (Text value : values) {
multipleOutputs.write(key.toString(), key, value);
}
}
}
catch(Exception e){ System.out.println("Caught Exception: " + e.getMessage());}
}
3) change number of reducers to 2 to get exactly 2 files.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.