如何让每个Mapper类在hadoop中读取相同的一个文件

Question

In my hadoop job, except my input data files, I want each mapper class(the map method) to read a common file which I put in the hdfs. This file will be read into each mapper, and save the content in each mapper. So how to do it?

Answer 1

Depending on your needs there are different approaches:

• Read the file directly from HDFS in each mapper. This is only recommended when the common file is realatively small.
• Use CompositeInputFormat to read multiple files at once in each mapper at perform a so called map-side-join. Both files will be splitted and partitioned the same way.
• Add the file to a DistributedCache during job setup. The file will be stored on every node an can accessed by all mappers.