HDFS联合中的块池如何运行

Question

So I was reading Hadoop: The Definitive Guide. A sentence in this page was what got me confused. So I have created an image depicting each sentence.

The sentence says,

Under federation, each namenode manages a namespace volume (the black squares depict the namespace volume), which is made up of the metadata for the namespace, and a block pool (depicted by the dark grey rectangle) containing all the blocks for the files in the namespace. Namespace volumes are independent of each other (in the image they are individual to each name node, shared with none), which means namenodes do not communicate with one another, and furthermore the failure of one namenode does not affect the availability of the namespaces managed by other namenodes. Block pool storage is not partitioned (and hence shared between all in the image), however, so datanodes register with each namenode in the cluster (again shared with all namenodes) and store blocks from multiple block pools (My question is how then we have more than one block pool? Doesn't the whole paragraph summarize that all the name nodes have meta data pointing to each block and therefore share a block pool?).

I am damn confused!

Answer 1

Your representation is inaccurate regarding the "Block pool" rectangle, it should read "Block pools".

I think it's worth looking at another representation:

So basically each block pools are managed independently from one another, each one is a set of blocks that belong to a single namespace. The Namenodes don't talk to each other which makes sense.

The reason behind this from what I've read is that this allows a namespace to generate Block IDs for new blocks without the need for coordination with the other namespaces. The failure of a namenode does not prevent the datanode from serving other namenodes in the cluster.

Answer 2

Just to get more clarity - in case NameNode NN-n in above diagram goes down,Pool-n also will be unavailable. So the datanode blocks maintained in Pool-n will be inaccessible until the Namenone NN-n is restored. OR it happens otherwise

Answer 3

I found this helpful, it from book hadoop-operations:
As first glance, it doesn't seem that federation is different from simply having multiple discreet clusters, save for the client plugin to view them as a single logical namespace. One of the major differentiating factors, however, is that each datanode in a federated cluster stores blocks for each namenode. When each namenode is formated, it generates a block pool in which block data associated with that namenode is stored. Each datanode, in turn, stores data for multiple block pools, and communicates with each namenode. When a namenode receives a heartbeat from a datanode, it learns about the total space on the datanode consumed by other block pools, as well as non-HDFS data. The rationale behind having all datanodes participate in all block pools rather than simply having discreet clusters is that this achieves better total utilization of datanode capacity. Instead, if we were to have a separate set of datanodes entirely for the heavily used namenode A, datanodes for namenode B would be underutilized while namenode A datanodes struggled to keep up with load.