数据节点之间的通信如何在Hadoop集群中工作？

Question

I am new to Hadoop and help with this questions is appreciated.

The replication of blocks in a cluster is handled by individual data nodes having a copy of the block, but how does this transfer take place without considering namenode.

I found that ssh is setup from slaves to master and master to slaves unlike slave to slave.

Could someone explain?

Is it through hadoop data transfer protocol like Client to DN communication ?

http://blog.cloudera.com/blog/2013/03/how-to-set-up-a-hadoop-cluster-with-network-encryption/

Answer 1

After digging into hadoop source code,I find datanodes use BlockSender class to transfer block data.Actually Socket is under the hood.

Below is my hack way to find this.(hadoop version 1.1.2 used here)

• DataNode Line 946 is offerService method, which is a main loop for service.

codes above is datanode send heartbeat to namenode mainly to tell it is alive.the return value are some commands which datanode will process.this is where block copy happens.

• digging into processCommand we come at Line 1160

here is a comment which we can be undoubtedly sure transferBlocks is what we want.

• digging into transferBlocks, we come at Line 1257, a private method.At the end of the method,

new Daemon(new DataTransfer(xferTargets, block, this)).start();

so,we know datanode start a new thread to do block copy.

• Look at DataTransfer in Line 1424,check at run method. at the nearly end of run method,we find following snippets:

// send data & checksum blockSender.sendBlock(out, baseStream, null);

from code above, we can know BlockSender is the actual worker.

I have done my work,It is up to you to find more,such as BlockReader

Answer 2

Whenever a block has to be written in HDFS, the NameNode will allocate space for this block on any datanode. It will also allocate space on other datanodes for the replicas of this block. Then it will instruct the first datanode to write the block and also to replicate the block on the other datanodes where space was allocated for the replicas.