在Java中连接到AWS EMR集群上的HBase

Question

I'm trying to access HBase on an AWS EMR cluster but the only thing I got is a :

2018-04-24 18:53:29 WARN  org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Operation timed out
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

I am trying to access it this way :

import com.google.protobuf.ServiceException;
import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.HBaseAdmin;

import java.io.IOException;

public class HBaseEMR {

    public static void main(String[] args) throws ServiceException, IOException {
        Configuration conf = HBaseConfiguration.create();
        conf.set("hbase.zookeeper.quorum","ec2-123456789101112.us-west-2.compute.amazonaws.com");
        conf.set("hbase.zookeeper.property.clientPort","2181");
        conf.set("hbase.master","ec2-123456789101112.us-west-2.compute.amazonaws.com");
        conf.set("hbase.master.port","60000");
        conf.set("hbase.rootdir", "s3://bucket/");

        HBaseAdmin.checkHBaseAvailable(conf);
    }
}

I modified the security groups to make an ssh connection through port 22 available but it's still not working. All advices welcome!

EDIT: After adding the port 2181 to the security group rules of the elasticmapreduce master, I have the following error :

2018-04-25 15:51:28 INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server ec2-<myDNS>.us-west-2.compute.amazonaws.com/<myDNS>:2181. Will not attempt to authenticate using SASL (unknown error)
2018-04-25 15:51:28 INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to ec2-<myDNS>.us-west-2.compute.amazonaws.com/<myDNS>:2181, initiating session
2018-04-25 15:51:28 INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server ec2-<myDNS>.us-west-2.compute.amazonaws.com/<myDNS>:2181, sessionid = 0x162f882
2018-04-25 15:51:30 INFO  org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation - Closing zookeeper sessionid=0x162f88235d80010
2018-04-25 15:51:30 INFO  org.apache.zookeeper.ZooKeeper - Session: 0x162f88235d80010 closed
2018-04-25 15:51:30 INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down
Exception in thread "main" org.apache.hadoop.hbase.MasterNotRunningException: java.net.UnknownHostException: ip-172-31-11-153.us-west-2.compute.internal
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1561)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1581)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1738)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.isMasterRunning(ConnectionManager.java:948)
    at org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:3162)
    at twitter.HBaseEMR.main(HBaseEMR.java:33)

Answer 1

Please Have a Look at this code. I have used this to connect to HBase. I think you are missing "zookeeper.znode.parent" Property.

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.log4j.Logger;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import com.naresh.common.util.PropertyUtil;



/**
 * 
 * @author Naresh Bharadwaj
 * This class contains functionality for Creating Hbase Connection
 * 
 */
@Component
public class HbaseConnection {

    private static final Logger _LOGGER = Logger.getLogger(HbaseConnection.class);

    @Autowired
    private PropertyUtil propertyUtil;

    private Boolean isKerberosEnabled;

    public Connection getConnection() throws IOException {

        Configuration conf = HBaseConfiguration.create();

                //Base 
        conf.set("hbase.zookeeper.quorum", propertyUtil.getValue("hbase.zookeeper.quorum"));
        conf.set("hbase.zookeeper.property.clientPort", propertyUtil.getValue("hbase.zookeeper.property.clientPort"));
        conf.set("zookeeper.znode.parent", propertyUtil.getValue("hbase.zookeeper.znode.parent"));


        return ConnectionFactory.createConnection(conf);
    }

    public void closeConnection(Connection connection) {
        try {
            connection.close();
        } catch (IOException e) {
            _LOGGER.error("Error while closing connection ", e);
        }
    }

}