如何将文件从HDFS复制到远程HDFS

Question

I want to copy files from my Hadoop cluster to the remote cluster.

I have the hadoop_conf file in the remote cluster and can access it by setting HADOOP_CONF_DIR.

I know the IP and port of the remote name node.

I want to copy the file through the namespace as ex below.

ex) hadoop fs -cp hdfs://MyNamespace/path/file hdfs://RemoteNamespace/path/file

However, if I do not configure hadoop_conf_dir , I do not know the remote namespace, and if I set the information of remote cluster in hadoop_conf_dir , I can not access my cluster's namespace.

Please let me know how to do it.

Answer 1

The typical way to copy between clusters is using distcp .

$ hadoop distcp hdfs://nn1:8020/foo/bar hdfs://nn2:8020/bar/foo

See DistCp Version2 Guide for more information.