Hadoop：每个tar / zip文件一个映射器

Question

I have several directories on which I want to compute statistics. ie my mapper function takes as input one folder tree and spits out some statics based on the contents of the directory and all its sub-directories. The computation takes a long time on each directory. There is no reducer.

I can create one tar/zip file for each directory I want to process and copy it into HDFS. But how do I ensure that a mapper will be created for each tar file and the entire contents of the tar file be sent to that mapper (so that I can traverse the contents of the tar file and generate statistics for that file)?

I would prefer to do this in Hadoop Streaming if possible. Is it possible to do this?

Answer 1

I take it you have a number of tar/zip files in HDFS as the input to your map/reduce job?

In that case you'll have to implement your own InputFormat to handle these. The input format implementation ( getSplits() ) determines the number of splits , each of those gets an individual mapper. So if you just return a single split for each input file, you'll be all set.

As far as I can see in the documentation, nothing in Hadoop Streaming prevents you from specifying your own InputFormat; this requires you to write a Java class though. (The question is how the inputformat and the script-based mapper should interact: as far as I understand, Hadoop streaming requires the mapper to receive its input via stdin, ie you can't easily pass the tar file itself for the script to operate on.)