Cloudera hdfs another namenode already locked the storage directory

Question

I am running CDH-5.3.2-1.cdh5.3.2.p0.10 with ClouderaManager on Centos 6.6. My HDFS service was working on a Cluster. But I wanted to change the mounting point for the hadoop data. Yet without success, so I came with the idea to rollback all changes, but the previous configuration doesnt work what is discouraging. I have two nodes within the cluster. One node for data is bad DataNodes Health Bad . In the log I have got a few errors

1:40:10.821 PM  ERROR   org.apache.hadoop.hdfs.server.common.Storage    
It appears that another namenode 931@spark1.xxx.xx has already locked the storage directory
1:40:10.821 PM  INFO    org.apache.hadoop.hdfs.server.common.Storage    
Cannot lock storage /dfs/nn. The directory is already locked
1:40:10.821 PM  WARN    org.apache.hadoop.hdfs.server.common.Storage    
java.io.IOException: Cannot lock storage /dfs/nn. The directory is already locked
1:40:10.822 PM  FATAL   org.apache.hadoop.hdfs.server.datanode.DataNode     
Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to spark1.xxx.xx/10.10.10.10:8022. Exiting. 
java.io.IOException: All specified directories are failed to load.
at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:463)
at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1318)
at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1288)
at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:320)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:221)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:829)
at java.lang.Thread.run(Thread.java:745)

I have been trying many possible solutions but without any luck.

• formatting hadoop namenode -format
• stopping cluster and rm -rf /dfs/* [and reformatting]
• some adjustments to /dfs/nn/current/VERSION file
• removing in_use.lock file and starting only a lacking node
• removing a file in /tmp/hsperfdata_hdfs/ with name like the pid locking the directory.

There are files in the directory

[root@spark1 dfs]# ll
total 8
drwxr-xr-x 3 hdfs hdfs   4096 Apr 28 13:39 nn
drwx------ 3 hdfs hadoop 4096 Apr 28 13:40 snn

There is no dn dir what is a bit interesting.

All operations on hdfs files I perform as an hdfs user. In the file /etc/hadoop/conf/hdfs-site.xml there is

  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:///dfs/nn</value>
 </property>

Answer 1

Here is a similar thread of CDH users google group which might help you : https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/FYu0gZcdXuE

Also did you do the namenode format from cloudera manager or command line ? Ideally you should be doing it through cloudera manager and not command line.