将本地目录的内容复制到hdfs中的目录

Question

I have got a requirement that I should be able to copy the contents of a directory from local system to a directory on HDFS.

The condition is only the directory contents should be copied to the location I have specified, not the source directory itself. Using command copyfromlocal I can achieve this. But I need to use Java. There is this method copyFromLocalFile which should be used for making a copy from local file system, the problem is it copies the directory itself. Also tried using FileUtils.copy method, gives the same result as the copyFromLocalFile

As a test I tried to copy directory contents from a test directory to another directory , both on the local file system. I used FileUtils.copyDirectory . This works but I cannot use it for HDFS. I have seen many links to related to this same question but could not find any way.

Guys, could you please let me know if this is possible or not or is it some design flaw? If this is possible how can I proceed ?

Answer 1

Ya it's really hard to get things done with FileSystem api exactly the way you want. Insufficient documentation makes things even harder(Too many methods, little explanation). I faced the same problem an year ago.

The only solution I got is iterative :

void copyFilesToDir(String srcDirPath, FileSystem fs, String destDirPath)
        throws IOException {
    File dir = new File(srcDirPath);
    for (File file : dir.listFiles()) {
        fs.copyFromLocalFile(new Path(file.getPath()),
                new Path(destDirPath, file.getName()));
    }
}

I think the code needs little explanation.