hadoop输入路径指定文件夹范围

Question

How do I specify a generic input path for my map reduce.

Example folder structure is:

   folderA/folderB/folderC/mainfolder/date/day/data files

There are many date folders and many days folders.

I want to drill down within a specific range of date folders folder and then pick up specific range of data files. If I try

'folderA/folderB/folderC/mainfolder/*/*' 

This will read all files. I want to specify a date forlder range ie read all files within 13-06-01 and 13-06-25 and and ignore all other date folders. How do I do that?

Answer 1

If you are providing

'folderA/folderB/folderC/mainfolder/*/*' 

as an input and want to filter out specific paths, you might want to create a custom PathFilter

In FileInputFormat you have this function-

---

static void setInputPathFilter (JobConf conf, Class<? extends PathFilter> filter)
Info: Set a PathFilter to be applied to the input paths for the map-reduce job

---

For eg

public static class CustomPathFilter implements PathFilter {
    @Override
    public boolean accept(Path path) {
        //you can implement your logic for finding the valid range of paths here.
        //The valid range of dates and days for directories and files can be input 
        //as arguments to the job.
        //Return true if you find a match or else return false.
        return false; 
    }
}

Register the PathFilter like this -

FileInputFormat.setInputPathFilter(job, CustomPathFilter.class);