Hbase regionserver getting shutdown

Question

We have installed a small Hadoop cluster with 3 machines. We are using Hortonworks.

We use the following code to connect to Hbase from java client.

Configuration config = HBaseConfiguration.create()
config.set(“hbase.zookeeper.quorum”, “xxxxxx”);
config.set(“hbase.zookeeper.property.clientPort”, “2181”);
config.set(“hbase.client.operation.timeout”, “1000”);
HConnection connection =HConnectionManager.createConnection(config);
HTableInterface hTable = connection.getTable(tableName);

But somehow the Hbase regionservers automatically shuts down after some time giving the following error.

2015-02-23 03:38:49,472 WARN [regionserver60020] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=ip-10-174-18-85.ap-southeast-1.compute.internal:2181,ip-10-149-130-178.ap-southeast-1.compute.internal:2181,ip-10-174-28-180.ap-southeast-1.compute.internal:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/ip-10-174-28-180.ap-southeast-1.compute.internal,60020,1424680062858
2015-02-23 03:38:49,472 ERROR [regionserver60020] zookeeper.RecoverableZooKeeper: ZooKeeper delete failed after 4 attempts
2015-02-23 03:38:49,472 WARN [regionserver60020] regionserver.HRegionServer: Failed deleting my ephemeral node
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/ip-10-174-28-180.ap-southeast-1.compute.internal,60020,1424680062858
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:156)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1270)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1259)
at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1286)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1016)
at java.lang.Thread.run(Thread.java:744)
2015-02-23 03:39:10,233 INFO [regionserver60020] regionserver.HRegionServer: stopping server ip-10-174-28-180.ap-southeast-1.compute.internal,60020,1424680062858; zookeeper connection closed.
2015-02-23 03:39:10,233 INFO [regionserver60020] regionserver.HRegionServer: regionserver60020 exiting
2015-02-23 03:39:10,236 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:66)
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:85)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2403)
2015-02-23 03:39:10,240 INFO [Thread-11] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@abde0e7

I tried increasing the JVM Heap size of regionserver to 2GB and session time out to 60s. But nothing seems to help.

Please help to get the issue resolved??

Answer 1

I cleared the zookeeper data once and its working fine now. I think some of the data inside the zookeeper was not consistent.

Answer 2

Exception shows "Failed Delete my Ephermal Code"

Solution: Those three machines have different timings. For example: machine1 time is slightly differs other machines leads above exception

If those three machines in the same date&time,this error will be resolved.

http://mail-archives.apache.org/mod_mbox/hbase-user/201305.mbox/%3CCAKxWWm1keFKNPmypLSq39RnTf8h=S5SFGyEY+4NB8D7ZOuqbxg@mail.gmail.com%3E