DataNode automatically getting restarted in the CDH5 cluster

Question

We have setup a cluster with 6 slave nodes. I am trying to see how replication happens when one of the DataNode dies.

I logged into one of the slave and killed the DataNode using the kill -9 command. After sometime the DataNode is restarted automatically and HDFS gets back into healthy status. I am verify this because the PID of the DataNode has changed.

I don't see any documentation on the above behavior of DataNode. Is this the Apache Hadoop or Cloudera CDH feature? Any reference to the documentation is appreciated.

Answer 1

As the pid of datanode has been changed, I don't think it is a behavior of datanode. If you are managing your cluster using Cloudera Manager, there is an option for restarting datanode daemon if it fails(Automatically Restart Process). This option will be set by default. When the datanode process gets failed or killed, As Automatic restart option is set Cloudera Scm agent will start the the datanode daemon.

For Automatic restart option : Choose HDFS services -> go to Configuration section -> Search for automatic restart .

This feature is available in CM 4.X release as well.