Map-Reduce作业无法传递预期的分区文件

Question

In a Map-Reduce job, I am using five different files where in my dataset contains values under two categories P and I . After I specific values are found, I am passing those into I-part-r-00000 file and accordingly, for P. I am using MultipleOutputformat class in reducer to achieve this.

My Mapper class contains:

public class parserMapper extends Mapper<LongWritable, Text, Text, Text> {
   public void map(LongWritable key, Text value, Context context)
   throws IOException, InterruptedException {

   String IPFLAG = "";
   String[] element_data= value.toString.split(","); 

    if(element_data[0].toString().trim().equalsIgnoreCase("005010X222A1")){
        IPFLAG = "P"; 
     }

    else {
       IPFLAG = "I";
     }

   if (IPFLAG == "P") {
     context.write(new Text(IPFLAG), new Text(theData));
     } 

   else if (IPFLAG == "I") {
   context.write(new Text(IPFLAG), new Text(theData));
     }

   else{
   System.out.println("No category found");
     } 


  }

  public void run(Context context) throws IOException, InterruptedException {
        setup(context);
        while (context.nextKeyValue()) {
            map(context.getCurrentKey(), context.getCurrentValue(), context);
        }
        cleanup(context);
    }


}

Reducer class includes:

public class parserReducer extends Reducer<Text, Text, Text, Text> {

    private MultipleOutputs multipleOutputs;

    @Override
    protected void setup(Context context) throws IOException, InterruptedException {
        multipleOutputs = new MultipleOutputs(context);
    }

    @Override
    protected void cleanup(Context context) throws IOException, InterruptedException {
        multipleOutputs.close();
    }

    @Override
    public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {

        Object c = null;

        try{
            if (!(key.toString().isEmpty())) {

                for (Text value : values) {

                    multipleOutputs.write(c, value, key.toString());
                }

            }
        }
        catch(Exception e){ System.out.println("Caught Exception: " + e.getMessage());}
    }


}

and Driver code includes =>

public class parserDriver {

 public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    conf.set("textinputformat.record.delimiter", "~"+"\n"+"ISA*");
    Job job = new Job(conf);
        job.setJobName("PARSER");
        job.setJarByClass(parserDriver.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(Text.class);
        job.setMapperClass(parserMapper.class);
        job.setReducerClass(parserReducer.class);
//      job.setOutputFormatClass(TextOutputFormat.class);
        LazyOutputFormat.setOutputFormatClass(job, TextOutputFormat.class);
    //  job.setOutputFormatClass(LazyOutputFormat.class);

/*       MultipleOutputs.addNamedOutput(job, "P", TextOutputFormat.class, Text.class, Text.class);
         MultipleOutputs.addNamedOutput(job, "I", TextOutputFormat.class, Text.class, Text.class);
*/
        // Pass as option -D mapred.reduce.tasks=<number>
        job.setNumReduceTasks(3);       

        /* This line is to accept the input recursively */
        //FileInputFormat.setInputDirRecursive(job, true);

        FileInputFormat.addInputPath(job, "/Users/Mohit/input");
        FileOutputFormat.setOutputPath(job, "/Users/Mohit/output");

        /*
         * Delete output file path if already exists
         */
        FileSystem fs = FileSystem.get(conf);

        if (fs.exists(outputFilePath)) {
            fs.delete(outputFilePath, true);
        }

        return job.waitForCompletion(true) ? 0: 1;
    }
}

Through all this, I am trying to achieve two partitions against a single file

file1 -> P-part-r00000, I-part-r00001

file2 -> P-part-r00002, I-part-r00003

. but I am getting two partitions against all the files being fed as input to this job.

file1, file2, file3, file4, file5 -> P-part-r00000, I-part-r00001

Not sure what am I missing here, if anybody can help please?

Answer 1

1) In your Driver add these lines to file naming:

   job.setOutputFormatClass(TextOutputFormat.class);
   MultipleOutputs.addNamedOutput(job, "I", TextOutputFormat.class,
          Text.class, Text.class);
   MultipleOutputs.addNamedOutput(job, "P", TextOutputFormat.class,
          Text.class, Text.class);

2) Change your reducer to send each value to file with specific name:

@Override
public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
    try{
        if (!(key.toString().isEmpty())) {

            for (Text value : values) {

                multipleOutputs.write(key.toString(), key, value);
            }

        }
    }
    catch(Exception e){ System.out.println("Caught Exception: " + e.getMessage());}
}

3) change number of reducers to 2 to get exactly 2 files.