Hadoop Mapreduce：Reducer的值是相反的顺序

Question

I will be doing the following in a much bigger file. for now,I have an example input file with the following values.

1000,SMITH,JERRY
1001,JOHN,TIA
1002,TWAIN,MARK
1003,HARDY,DENNIS
1004,CHILD,JACK
1005,CHILD,NORTON
1006,DAVIS,JENNY
1007,DAVIS,KAREN
1008,MIKE,JOHN
1009,DENNIS,SHERIN

now what i am doing is running a mapreduce job to encrypt the last name of each record and write back an output. and i am using the mapper partition number as the key and the modified text as value.

so the output from mapper will be,

0   1000,Mj4oJyk=,,JERRY
0   1001,KzwpPQ,TIA
0   1002,NSQgOi8,MARK
0   1003,KTIzNzg,DENNIS
0   1004,IjsoPyU,JACK
0   1005,IjsoPyU,NORTON
0   1006,JTI3OjI,JENNY
0   1007,JTI3OjI,KAREN
0   1008,LDoqNg,JOHN
0   1009,JTYvPSgg,SHERIN

I don't want any sorting to be done.I also use a reducer because, in case of a larger file, there will be multiple mappers and if no reducer, multiple output files will be written. so i use a single reduce to merge values from all mappers and write to single file. now the input values to reducer comes in reversed order and in the order from mapper. it is like the following,

1009,JTYvPSgg,SHERIN
1008,LDoqNg==,JOHN
1007,JTI3OjI=,KAREN
1006,JTI3OjI=,JENNY
1005,IjsoPyU=,NORTON
1004,IjsoPyU=,JACK
1003,KTIzNzg=,DENNIS
1002,NSQgOi8=,MARK
1001,KzwpPQ==,TIA
1000,Mj4oJyk=,JERRY

Why is it reversing the order? and how can i maintain the same order from mapper? any suggestions will be helpfull

EDIT 1 :

the Driver code is,

Configuration conf = new Configuration();
Job job = Job.getInstance(conf);
    job.setJobName("encrypt");
    job.setJarByClass(TestDriver.class);
    job.setMapperClass(TestMap.class);
    job.setNumReduceTasks(1);
    job.setReducerClass(TestReduce.class);
    job.setMapOutputKeyClass(IntWritable.class);
    job.setMapOutputValueClass(Text.class);
     job.setOutputKeyClass(Text.class);
     job.setOutputValueClass(IntWritable.class);

    FileInputFormat.addInputPath(job, new Path(hdfsInputPath));
    FileOutputFormat.setOutputPath(job, new Path(hdfsOutputPath));
System.exit(job.waitForCompletion(true) ? 0 : 1);

the mapper code is,

        inputValues = value.toString().split(",");
        stringBuilder = new StringBuilder();
        TaskID taskId = context.getTaskAttemptID().getTaskID();
        int partition = taskId.getId();

 // the mask(inputvalue) method is called to encrypt input values and write to stringbuilder in appropriate format
        mask(inputvalues);
        context.write(new IntWritable(partition), new Text(stringBuilder.toString()));

The reducer code is,

       for(Text value : values) {
        context.write(new Text(value), null);
       }

Answer 1

The base idea of MapReduce is that the order in which things are done is irrelevant. So you cannot (and do not need to) control the order in which

• the input records go through the mappers.
• the key and related values go through the reducers.

The only thing you can control is the order in which the values are placed in the iterator that is made available in the reducer.

For that you can use the Object key to maintain the order of values. The LongWritable part (or the key) is the position of the line in the file (Not line number, but position from start of file). You can use that part to keep track of which line was first.

Then your mapper code will be changed to

protected void map(Object key, Text value, Mapper<Object, Text, LongWritable, Text>.Context context)
        throws IOException, InterruptedException {
    inputValues = value.toString().split(",");
    stringBuilder = new StringBuilder();
    mask(inputValues);
    // the mask(inputvalue) method is called to encrypt input values and write to stringbuilder in appropriate format
    context.write(new LongWritable(((LongWritable) key).get()), value);

}

Note: you can change all IntWritable to LongWritable in your code but be careful.

Answer 2

    inputValues = value.toString().split(",");
    stringBuilder = new StringBuilder();
    TaskID taskId = context.getTaskAttemptID().getTaskID();
    //preserve the number value for sorting
    IntWritable idNumber = new IntWritable(Integer.parseInt(inputValue[0])

    // the mask(inputvalue) method is called to encrypt input values and write to stringbuilder in appropriate format
    mask(inputvalues);
    context.write(idNumber, new Text(stringBuilder.toString()));

I made some assumptions because you did not have the full code of the mapper. I assumed that inputValues was a string array due to the toString() output. The first value of the array should be the number value from your input, however it is now a string. You must convert the number back to IntWritable to match what your mapper is emitting IntWritable,Text . The hadoop framework will sort by key and with the key being of type IntWritable it will sort in ascending order. The code you provided is using the task ID and from reading the API https://hadoop.apache.org/docs/r2.4.1/api/org/apache/hadoop/mapred/TaskAttemptID.html#getTaskID() It was unclear whether this would provide an order to your values as you desired. To control the order of output I would recommend using the first value of your string array and convert to IntWritable. I don't know if this violates your intent to mask the inputValues .

EDIT

To follow up with your comment. You can simply multiply the partition by -1 this will cause the hadoop framework to reverse the order.

int partition = -1*taskId.getId();