将文件/块从 HDFS 复制到从节点的本地文件系统

Question

In Hadoop, I understand that the master node(Namenode) is responsible for storing the blocks of data in the slave machines(Datanode).

When we use -copyToLocal or -get , from the master, the files could be copied from the HDFS to the local storage of the master node. Is there any way the slaves can copy the blocks(data) that are stored in them, to their own local file system?

For ex, a file of 128 MB could be split among 2 slave nodes storing 64MB each. Is there any way for the slave to identify and load this chunk of data to its local file system? If so, how can this be done programmatically? Can the commands -copyToLocal or -get be used in this case also? Please help.

Answer 1

Short Answer: No

The data/files cannot be copied directly from Datandode 's. The reason is, Datanodes store the data but they don't have any metadata information about the stored files. For them, they are just block of bits and bytes. The metadata of the files is stored in the Namenode . This metadata contains all the information about the files (name, size, etc.). Along with this, Namenode keeps track of which blocks of the file are stored on which Datanodes . The Datanodes are also not aware of the ordering of the blocks, when actual files are splits in multiple blocks.

Answer 2

Can the commands -copyToLocal or -get be used in this case also?

Yes, you can simply run these from the slave. The slave will then contact the namenode (if you've configured it properly) and download the data to your local filesystem.

What it doesn't do is a "short-circuit" copy, in which it would just copy the raw blocks between directories. There is also no guarantee it will read the blocks from the local machine at all, as your commandline client doesn't know its location.

Answer 3

HDFS blocks are stored on the slaves local FS only . you can dig down the directory defined under property "dfs.datanode.dir" But you wont get any benefit of reading blocks directly (without HDFS API). Also reading and editing block files directory can corrupt the file on HDFS.

If you want to store data on different slave local then you will have to implement your logic of maintaining block metadata (which is already written in Namenode and do for you).

Can you elaborate more why you want to distribute blocks by yourself when Hadoop takes care of all challenges faced in distributed data?

Answer 4

You can copy particular file or directory from one slave to another slave by using distcp

Usage: distcp slave1address slave2address