Is there a way to add nodes to a running Hadoop cluster?

Question

I have been playing with Cloudera and I define the number of clusters before I start my job then use the cloudera manager to make sure everything is running.

I'm working on a new project that instead of using hadoop is using message queues to distribute the work but the results of the work are stored in HBase. I might launch 10 servers to process the job and store to Hbase but I'm wondering if I later decided to add a few more worker nodes can I easily (read: programmable) make them automatically connect to the running cluster so they can locally add to clusters HBase/HDFS?

Is this possible and what would I need to learn in order to do it?

Answer 1

Here is the documentation for adding a node to Hadoop and for HBase . Looking at the documentation, there is no need to restart the cluster. A node can be added dynamically.

Answer 2

Following steps should help you launch the new node into the running cluster.

1> Update the /etc/hadoop/conf/slaves list with the new node-name
2> Sync the full configuration /etc/hadoop/conf to the new datanode from the Namenode. If the file system isn't shared.  
2>  Restart all the hadoop services on Namenode/Tasktracker and all the services on the new Datanode. 
3>  Verify the new datanode from the browser http://namenode:50070
4>  Run the balancer script to readjust the data between the nodes. 

If you don't want to restart the services on the NN, when you add a new node. I would say add the names ahead to the slaves configuration file. So they report as decommission/dead nodes until they are available. Following the above DataNode only steps. Again this not the best practice.

Answer 3

If I understand you correctly you have workers you coordinate by yourself that connect to HBase to save their data. you can have as many of those as you need and they can connect to Hbase as they're added (as long as they can see the zookeeper quorum).

If you are talking about increasing the Hadoop cluster. Since you already use Cloudera you can do that via the cloudera Manager REST API or the the Java client someone implemented for it

Answer 4

It can be done without restarting the hadoop cluster. As per this document , You can add nodes in the includes file and and making some changes in the hdfs-site.xml and mapred-site.xml files, you can achieve this. Detailed instructions are given in the this document

Answer 5

Updated Answer for Cloudera using CDH 5.8.5 (Hadoop 2.6)-

To add a new node to your cluster, follow these steps on ClouderaManager UI,

1. Click on your cluster name.
2. Go to Hosts List.
3. Once on the hosts page, click 'Add New Hosts to Cluster'.
4. Enter the IP of your host and Search.
5. Keep following the instructions and continue to next steps.
6. Finally assign roles to your new node, for example if it's a data-node,assign only datanode related roles and continue.
7. Finally your new node is added to your cluster. click Finish.