通过停留在zookeeper上的java客户端连接到hbase 1.0.3 ClientCnxn：服务器上的会话建立完成

Question

I am stuck on connecting to hbase 1.0.3 via simple java client. The program hangs at:

[main] zookeeper.ZooKeeper: Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=90000 watcher=hconnection-0x1e67b8720x0, quorum=127.0.0.1:2181, baseZNode=/hbaseenter code here
[main-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
[main-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session
[main-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x153d8383c530008, negotiated timeout = 40000

The code is very simple:

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.client.Table;

public class ScanAllTables {

public static void main(String args[]) throws IOException{

    // Instantiating Configuration class
    Configuration config = HBaseConfiguration.create();

    System.out.println(config.get("hbase.rootdir"));
    System.out.println(config.get("hbase.zookeeper.property.dataDir"));
    System.out.println(config.get("hbase.zookeeper.quorum"));
    System.out.println(config.get("hbase.zookeeper.property.clientPort"));

    Connection connection = ConnectionFactory.createConnection(config);

    //System.out.println(connection);

    // Instantiating Table class
    Table  table = connection.getTable(TableName.valueOf(HBaseTables.APPLICATION_TRACE_INDEX));

    //System.out.println(table.getName());

    // Instantiating the Scan class
    Scan scan = new Scan();

     // Scanning the required columns
    //scan.addFamily(HBaseTables.APPLICATION_TRACE_INDEX_CF_TRACE);
    //scan.setId("ApplicationTraceIndexScan");

    // Getting the scan result
    ResultScanner scanner = table.getScanner(scan);

    // Reading values from scan result
    for (Result result = scanner.next(); result != null; result = scanner.next()) {
        System.out.println("Found row : " + result);
    }


    //closing the scanner
    scanner.close();

    table.close();
        connection.close();
     }

}

The jars I am using are:

commons-collections-3.2.1.jar
commons-configuration-1.6.jar
commons-lang-2.6.jar
commons-logging-1.2.jar
guava-12.0.1.jar
hadoop-auth-2.5.1.jar
hadoop-client-2.5.1.jar
hadoop-common-2.5.1.jar
hbase-client-1.0.3.jar
hbase-common-1.0.3.jar
hbase-hadoop-compat-1.0.3.jar
hbase-hadoop2-compat-1.0.3.jar
hbase-it-1.0.3.jar
hbase-protocol-1.0.3.jar
hbase-resource-bundle-1.0.3.jar
hbase-rest-1.0.3.jar
htrace-core-3.0.4.jar
htrace-core-3.1.0-incubating.jar
log4j-1.2.17.jar
netty-3.2.4.Final.jar
netty-all-4.0.23.Final.jar
protobuf-java-2.5.0.jar
slf4j-api-1.7.7.jar
slf4j-log4j12-1.7.7.jar
zookeeper-3.4.6.jar

My hbase-site.xml is

<configuration>
  <property>
    <name>hbase.rootdir</name>
    <value>file:///path/to/data/hbase</value>
  </property>
  <property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>file:///path/to/data/zookeeper</value>
  </property>
  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>127.0.0.1</value>
  </property>
</configuration>

In my hosts I have this entry: 127.0.0.1 localhost.localdomain localhost Sachin-PC

Also in hbase regionservers has localhost I have tried many options for hbase.zookeeper.quorum like localhost, Sachin-PC but none have worked.

Also note the jars I am using are of same version in hbase lib.

I am just not able to wrap my head around as what is causing the failure to scan table for this simple code.

Note the code stops at line ResultScanner scanner = table.getScanner(scan); So connection is happening but it is not getting any data.

Please help!

Thanks Sachin

ps:

    DEBUG [hconnection-0x1e67b872-shared--pool1-t1] ipc.AbstractRpcClient: Use SIMPLE authentication for service ClientService, sasl=false                             
    DEBUG [hconnection-0x1e67b872-shared--pool1-t1] ipc.AbstractRpcClient: Connecting to Sachin-PC/127.0.0.1:55161                                                     
    DEBUG [hconnection-0x1e67b872-shared--pool1-t1] ipc.AbstractRpcClient: Use SIMPLE authentication for service ClientService, sasl=false                             
    DEBUG [hconnection-0x1e67b872-shared--pool1-t1] ipc.AbstractRpcClient: Not trying to connect to Sachin-PC/127.0.0.1:55161 this server is in the failed servers list

Please see these additional logs: As we can see the connection happens and region location detail is also fetched. But when it tries to connect to that region to actually fetch the table scan details we get the socket close error and the is then added to failed server list. This process repeats again and again and hence the process is stuck.

[main-EventThread] zookeeper.ZooKeeperWatcher: hconnection-0x1e67b872-0x153e135af570008 connected

[main] client.ZooKeeperRegistry: Looking up meta region location in ZK, connection=org.apache.hadoop.hbase.client.ZooKeeperRegistry@69b794e2

[main] client.ZooKeeperRegistry: Looked up meta region location, connection=org.apache.hadoop.hbase.client.ZooKeeperRegistry@69b794e2; serverName=sachin-pc,55964,1459772310378
[main] client.MetaCache: Cached location: [region=hbase:meta,,1.1588230740, hostname=sachin-pc,55964,1459772310378, seqNum=0]

[hconnection-0x1e67b872-shared--pool1-t1] ipc.AbstractRpcClient: Connecting to Sachin-PC/127.0.0.1:55964

java.net.SocketException: Socket is closed
    at sun.nio.ch.SocketAdaptor.getOutputStream(Unknown Source)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.closeConnection(RpcClientImpl.java:429)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleConnectionFailure(RpcClientImpl.java:477)


java.net.SocketException: Socket is closed
    at sun.nio.ch.SocketAdaptor.getInputStream(Unknown Source)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.closeConnection(RpcClientImpl.java:436)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleConnectionFailure(RpcClientImpl.java:477)



[hconnection-0x1e67b872-shared--pool1-t1] ipc.AbstractRpcClient: IPC Client (1890187342) connection to Sachin-PC/127.0.0.1:55964 from Sachin: marking at should close, reason: Connection refused: no further information

Answer 1

I figured out the issue. The region server was listening to 192.168.1.102:55964 and not 127.0.0.1:55964. 192.168.1.102 is the IP of my machine and Sachin-PC is my machine name.

In hosts my entry was 127.0.0.1 localhost.localdomain localhost Sachin-PC

I removed Sachin-PC from there and it works now. Client now connects to 192.168.1.102:55964.

I however don't understand one thing: in regionservers I have specified: 127.0.0.1 Also in my hbase-site.xml I specified hbase.zookeeper.quorum as 127.0.0.1

When I do netstat I see zookeeper, master, master web UI, region server web UI all listening at 0.0.0.0 However regionserver listening at 192.168.1.102 which is my machine's IP address.

So why is regionserver listening at a different IP.

One more thing, when I start hbase, I see no log file getting generated. How do I specify log file to be generated.