Hadoop映射器：行与文件

Question

I am learning Hadoop/mapreduce and have a question about various possibilities of splitting up mappers.

In the standard 'wordcount' scenario, each different process works on a unique line and does some basic math (addition). Is it possible, however, to have each process work on a unique file? For example, if I have 500,000 unique files, each of which is ~5M, can I tell Hadoop that each process should perform some analysis on each file and then perform statistics on the resulting analysis (for example, average the results together)?

For example, suppose each file contains:

{name}
{data1}
{data2}
...
{dataN}

and I want to perform a mathematical function on this file to get F({name}) = [value1, value2, value3] based on {data1, ..., dataN} , and, at the end, I want to find the average of all possible [value1, value2, value3] arrays for each {name} . In this case, if I use Hadoop to work on each line, it will not help since each data must be associated with a name , so I would like to have Hadoop maintain knowledge of which name it is working with.

If this is possible, would the calculation of F be the 'map' phase and then the averaging of [value1, value2, value3] arrays be the 'reduce' phase?

So, to consolidate the question into a clear one-liner: How can I get Hadoop to split up work on files, rather than lines?

Answer 1

We can get the filename and output that as mapper's output key. The mappers output value can be the value like value1,value2, value3 etc. The snippet to get the file name is as follows

InputSplit split = context.getInputSplit();
String fileName = split.getPath().getName();

In the reducer part we can iterate as per the key which is our filenames here and do the necessary operations like average,sum etc. The reducer output can have the filenames along with the value.

Answer 2

这可能是您正在寻找的内容： http : //wiki.apache.org/hadoop/FAQ#How_do_I_get_each_of_a_job.27s_maps_to_work_on_one_complete_input_file_and_not_allow_the_framework_to_split-up_the_files.3F