使用FileUtil API将文件复制到同一hdfs中会花费太多时间

Question

I have 1 HDFS and my local system from where I'm executing my program to perform a copy inside a same hdfs system. Like: hadoop fs -cp /user/hadoop/SrcFile /user/hadoop/TgtFile

I'm using:

FileUtil.copy(FileSystem srcFS,
FileStatus srcStatus,
FileSystem dstFS,
Path dst,
boolean deleteSource,
boolean overwrite,
Configuration conf) 

But something weird is happening, when I'm doing copy from command line, it just take a moment to copy but when I do it programmatically it takes a 10 - 15 minute to copy 190 mb file.

For me it look like it's streaming the data via my local system instead of streaming directly because the destination is also on the same filesystem as of source.

Correct me if I'm wrong and also help me to find out the best solution.

Answer 1

You are right in that using FileUtil.copy the streaming is passed through your program (src --> yourprogram --> dst). If hadoops filesystem shell API (hadoop dfs -cp) is faster than you can use the same through Runtime.exec(cmd)

https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java