HBase Java Client does not work (MasterNotRunningException exception )

Question

I'm trying to write a remote HBase client using Java. Here is the code for reference :

package ttumdt.app.connector;


import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.MasterNotRunningException;
import org.apache.hadoop.hbase.ZooKeeperConnectionException;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.filter.BinaryComparator;
import org.apache.hadoop.hbase.filter.CompareFilter;
import org.apache.hadoop.hbase.filter.Filter;
import org.apache.hadoop.hbase.filter.FilterList;
import org.apache.hadoop.hbase.filter.RowFilter;
import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
import org.apache.hadoop.hbase.util.Bytes;

import java.io.IOException;
import java.util.Arrays;
import java.util.HashSet;
import java.util.Set;

public class HBaseClusterConnector {
    private final String MASTER_IP = "10.138.168.185";
    private final String ZOOKEEPER_PORT = "2181";

    final String TRAFFIC_INFO_TABLE_NAME = "TrafficLog";
    final String TRAFFIC_INFO_COLUMN_FAMILY = "TimeStampIMSI";

    final String KEY_TRAFFIC_INFO_TABLE_BTS_ID = "BTS_ID";
    final String KEY_TRAFFIC_INFO_TABLE_DATE = "DATE";
    final String COLUMN_IMSI = "IMSI";
    final String COLUMN_TIMESTAMP = "TIME_STAMP";

    private final byte[] columnFamily = Bytes.toBytes(TRAFFIC_INFO_COLUMN_FAMILY);
    private final byte[] qualifier= Bytes.toBytes(COLUMN_IMSI);

    private Configuration conf = null;

    public HBaseClusterConnector () throws MasterNotRunningException, ZooKeeperConnectionException {
        conf = HBaseConfiguration.create();
        conf.set("hbase.zookeeper.quorum",MASTER_IP);
        conf.set("hbase.zookeeper.property.clientPort",ZOOKEEPER_PORT);
        HBaseAdmin.checkHBaseAvailable(conf);
    }

    /**
     * This filter will return list of IMSIs for a given btsId and ime interval
     * @param btsId : btsId for which the query has to run
     * @param startTime : start time for which the query has to run
     * @param endTime : end time for which the query has to run
     * @return returns IMSIs as set of Strings
     * @throws IOException
     */
    public Set<String> getInfoPerBTSID(String btsId, String date,
                                       String startTime, String endTime)
            throws IOException {
        Set<String> imsis = new HashSet<String>();

        //ToDo : better exception handling
        HTable table = new HTable(conf, TRAFFIC_INFO_TABLE_NAME);
        Scan scan = new Scan();

        scan.addColumn(columnFamily,qualifier);
        scan.setFilter(prepFilter(btsId, date, startTime, endTime));

        // filter to build where timestamp

        Result result = null;
        ResultScanner resultScanner = table.getScanner(scan);

        while ((result = resultScanner.next())!= null) {
            byte[] obtainedColumn = result.getValue(columnFamily,qualifier);
            imsis.add(Bytes.toString(obtainedColumn));
        }

        resultScanner.close();

        return imsis;
    }

    //ToDo : Figure out how valid is this filter code?? How comparison happens
    // with eqaul or grater than equal etc


    private Filter prepFilter (String btsId, String date,
                               String startTime, String endTime)
    {
        byte[] tableKey = Bytes.toBytes(KEY_TRAFFIC_INFO_TABLE_BTS_ID);
        byte[] timeStamp = Bytes.toBytes(COLUMN_TIMESTAMP);

        // filter to build -> where BTS_ID = <<btsId>> and Date = <<date>>
        RowFilter keyFilter = new RowFilter(CompareFilter.CompareOp.EQUAL,
                new BinaryComparator(Bytes.toBytes(btsId+date)));

        // filter to build -> where timeStamp >= startTime
        SingleColumnValueFilter singleColumnValueFilterStartTime =
                new SingleColumnValueFilter(columnFamily, timeStamp,
                        CompareFilter.CompareOp.GREATER_OR_EQUAL,Bytes.toBytes(startTime));

        // filter to build -> where timeStamp <= endTime
        SingleColumnValueFilter singleColumnValueFilterEndTime =
                new SingleColumnValueFilter(columnFamily, timeStamp,
                        CompareFilter.CompareOp.LESS_OR_EQUAL,Bytes.toBytes(endTime));

        FilterList filterList = new FilterList(FilterList.Operator.MUST_PASS_ALL, Arrays
                .asList((Filter) keyFilter,
                        singleColumnValueFilterStartTime, singleColumnValueFilterEndTime));
        return filterList;
    }


    public static void main(String[] args) throws IOException {
        HBaseClusterConnector flt = new HBaseClusterConnector();
        Set<String> imsis= flt.getInfoPerBTSID("AMCD000784", "26082013","104092","104095");
        System.out.println(imsis.toString());
    }
}

I'm currently using Cloudera quick start VM to test this.

The problem is; if i run this very code on VM it works absolutely fine. But it fails with below error if it is run from outside. And I'm suspecting it has something to do with the VM setting rather than anything else. Please note that I've already checked if I can connect to the node manager / job tracker of the VM from host machine and it works absolutely fine. When I run the code from my host OS instead of running it on VM; I get the below error :

2013-10-15 18:16:04.185 java[652:1903] Unable to load realm info from SCDynamicStore
Exception in thread "main" org.apache.hadoop.hbase.MasterNotRunningException: Retried 1 times
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:138)
    at org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:1774)
    at ttumdt.app.connector.HBaseClusterConnector.<init>(HBaseClusterConnector.java:47)
    at ttumdt.app.connector.HBaseClusterConnector.main(HBaseClusterConnector.java:117)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)

Process finished with exit code 1

Please note that; the master node is actually running. The zookeper log shows that it has established connection with the host OS :

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

6:16:03.274 PM  INFO    org.apache.zookeeper.server.ZooKeeperServer     

Client attempting to establish new session at /10.138.169.81:50567

6:16:03.314 PM  INFO    org.apache.zookeeper.server.ZooKeeperServer     

Established session 0x141bc2487440004 with negotiated timeout 60000 for client /10.138.169.81:50567

6:16:03.964 PM  INFO    org.apache.zookeeper.server.PrepRequestProcessor    

Processed session termination for sessionid: 0x141bc2487440004

6:16:03.996 PM  INFO    org.apache.zookeeper.server.NIOServerCnxn   

Closed socket connection for client /10.138.169.81:50567 which had sessionid 0x141bc2487440004

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

But I see no trace of any activity in Master or RegionServer log.

Please note that my host OS is Mac OSX 10.7.5

As per the resource available; this should work fine; though some suggest the simple HBase java client never works. I'm confused; and eagerly waiting for pointers !!! Please reply

Answer 1

The HBase Java client certainly does work!

The most likely explanation is that your client can't see the machine that the master is running on for some reason.

One possible explanation is that, although you are connecting to Zookeeper using an IP address, the HBase client is attempting to connect to the master using its hostname.

So, if you ensure that you have entries in your hosts file (on the client) that match the hostname of the machine running the master, this may resolve the problem.

Check that you can access the master Web UI at <hostname>:60010 from your client machine.

Answer 2

start your hiveserver2 on different port and then try connecting

command to connect hiveserver2 on different port (make sure hive is in path ) :

hive --service hiveserver2 --hiveconf hive.server2.thrift.port=13000