简体   繁体   中英

Running two mapper on a input file and a reducer for simple hadoop mapreduce jobs

In a simple MapReduce program, there is one input file, that after splitting each line, it will be mapped. But now I want to read each line and after split, the line, map it two times by different keys. reading one line and two maps (two times using context.write). I read: Running two mapper and two reducer for simple hadoop mapreduce jobs That answered: So just put 2 files into your input directories so that you can get 2 mappers running. Now I should to put two same files?

There's no point to read line in mapper, split it and pass somewhere. Process it immediately:

void map(K key, Text value, Context ctx) {
    String k1 = getKey1(value);
    String k2 = getKey2(value);
    map1(k1, value);
    map2(k2, value);
}

void map1(...) { ... }
void map2(...) { ... }

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM