How to move files within the Hadoop HDFS directory?

Question

I need to move the files from one HDFS directory to another HDFS directory.

I wanted to check if there's some easier way (some HDFS API) to achieve the same task, other than InputStream/OutputStream ?

I've heard of FileSystem.rename(srcDir, destDir); but is unsure if this will delete the original src directory.

I don't want to remove the original directory structure, only move the files from one folder to another directory.

eg

input Dir - /testHDFS/input/*.txt
dest Dir - /testHDFS/destination

After moving the files, directory should look something like this :-

input Dir - /testHDFS/input
dest Dir - /testHDFS/destination/*.txt

PS : I want to achieve this working inside mapper function for each file.

Any help would be appreciated.

Answer 1

FileSystem.rename will move the file from source to destination directory. I believe you can use it for your requirement.

Answer 2

The best way to do this is with org.apache.hadoop.fs.FileUtil.copy() , setting the deleteSource parameter to true . People commonly use FileSystem.rename() , but that function will fail silently for invisible issues ( such as the source and destination Paths being on different volumes )

Answer 3

您可以使用DistCp以编程方式对此进行验证