hbase startup slow — Error while calling watcher java.lang.NullPointerException: ZK is null

Question

Recently establishing a connection to my local 0.94 hbase has been excruciatingly slow. I get the errors below. I think it started happening when I upgraded my mac to yosemite. I have tried all the easy fixes, restart, clear data/zk directories, reinstall hbase, etc., but nothing seems to help.

My configuration is very simple, just overriding hbase.rootdir , hbase.zookeeper.quorum , and hbase.zookeeper.property.dataDir . I also get messages like this:

 java[3323:30250] Unable to load realm info from SCDynamicStore

but those occurred previously without the exception stack/very slow connection.

ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw
java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:143)
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:127)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1506)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:714)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:787)
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:175)
    at com.feedly.normandy.persistence.hbase.PersistenceManager.init(PersistenceManager.java:244)
    at com.feedly.normandy.HBaseUnitTest.createPersistenceManager(HBaseUnitTest.java:152)
    at com.feedly.normandy.HBaseUnitTest.beforeClass(HBaseUnitTest.java:159)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

SEVERE: Error while calling watcher 
java.lang.NullPointerException: ZK is null
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:366)
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:303)
    at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)

Answer 1

GrepCode Link Here claims that ZookKeeperWatcher throws a NullPointerException because the underlying zooKeeper handle goes null during a connection related event. I would suggest doing a netstat on the ZK host and seeing if it's dropping your connection at some point. If you don't see that happening, download the source jar for this class and step into the connectionEvent() method to see exactly when the object goes null, it might lead you to the root cause.

Answer 2

pretty strange, but seems to be related to localhost lookups. I tried a few things from

How can I eliminate slow resolving/loading of localhost/virtualhost (a 2-3 second lag) on Mac OS X Lion?

until I could get lookups to happen quickly and then everything became snappy again.

Answer 3

I have the same problem on my machine. I solved it doing the following steps:

1. Open terminal and write the command: hostname - as the result you'll see the hostname of your PC. Copy this name.

2. Change the file /etc/hosts , adding hostname copied from previous step:
   127.0.0.1 localhost {put here the name}

Hope that this will help you too.