[英]Connect to HBase on AWS EMR cluster in java
我正在嘗試訪問AWS EMR集群上的HBase,但我唯一得到的是:
2018-04-24 18:53:29 WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Operation timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
我正在嘗試以這種方式訪問它:
import com.google.protobuf.ServiceException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import java.io.IOException;
public class HBaseEMR {
public static void main(String[] args) throws ServiceException, IOException {
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.zookeeper.quorum","ec2-123456789101112.us-west-2.compute.amazonaws.com");
conf.set("hbase.zookeeper.property.clientPort","2181");
conf.set("hbase.master","ec2-123456789101112.us-west-2.compute.amazonaws.com");
conf.set("hbase.master.port","60000");
conf.set("hbase.rootdir", "s3://bucket/");
HBaseAdmin.checkHBaseAvailable(conf);
}
}
我修改了安全組,以使通過端口22的ssh連接可用,但仍無法正常工作。 歡迎所有建議!
編輯:將端口2181添加到elasticmapreduce主節點的安全組規則后,我出現以下錯誤:
2018-04-25 15:51:28 INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server ec2-<myDNS>.us-west-2.compute.amazonaws.com/<myDNS>:2181. Will not attempt to authenticate using SASL (unknown error)
2018-04-25 15:51:28 INFO org.apache.zookeeper.ClientCnxn - Socket connection established to ec2-<myDNS>.us-west-2.compute.amazonaws.com/<myDNS>:2181, initiating session
2018-04-25 15:51:28 INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server ec2-<myDNS>.us-west-2.compute.amazonaws.com/<myDNS>:2181, sessionid = 0x162f882
2018-04-25 15:51:30 INFO org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation - Closing zookeeper sessionid=0x162f88235d80010
2018-04-25 15:51:30 INFO org.apache.zookeeper.ZooKeeper - Session: 0x162f88235d80010 closed
2018-04-25 15:51:30 INFO org.apache.zookeeper.ClientCnxn - EventThread shut down
Exception in thread "main" org.apache.hadoop.hbase.MasterNotRunningException: java.net.UnknownHostException: ip-172-31-11-153.us-west-2.compute.internal
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1561)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1581)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1738)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.isMasterRunning(ConnectionManager.java:948)
at org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:3162)
at twitter.HBaseEMR.main(HBaseEMR.java:33)
請看一下這段代碼。 我已經使用它來連接到HBase。 我認為您缺少“ zookeeper.znode.parent”屬性。
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.log4j.Logger;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import com.naresh.common.util.PropertyUtil;
/**
*
* @author Naresh Bharadwaj
* This class contains functionality for Creating Hbase Connection
*
*/
@Component
public class HbaseConnection {
private static final Logger _LOGGER = Logger.getLogger(HbaseConnection.class);
@Autowired
private PropertyUtil propertyUtil;
private Boolean isKerberosEnabled;
public Connection getConnection() throws IOException {
Configuration conf = HBaseConfiguration.create();
//Base
conf.set("hbase.zookeeper.quorum", propertyUtil.getValue("hbase.zookeeper.quorum"));
conf.set("hbase.zookeeper.property.clientPort", propertyUtil.getValue("hbase.zookeeper.property.clientPort"));
conf.set("zookeeper.znode.parent", propertyUtil.getValue("hbase.zookeeper.znode.parent"));
return ConnectionFactory.createConnection(conf);
}
public void closeConnection(Connection connection) {
try {
connection.close();
} catch (IOException e) {
_LOGGER.error("Error while closing connection ", e);
}
}
}
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.