SolrJ hanging when connecting to zookeeper

Question

I have a local two instance Solr Cloud setup with a single zookeeper instance. I am trying to connect via SolrJ to execute a query however my code hangs for 2mins or so when executing the query and then fails. I have followed the basic example on the Solr wiki. The logs/code is below

2016-07-24 13:29:01.932  INFO 83666 --- [qtp699221219-28] org.apache.zookeeper.ZooKeeper           : Initiating client connection, connectString=localhost:2181 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.SolrZkClient$3@496eab9
2016-07-24 13:29:01.948  INFO 83666 --- [qtp699221219-28] o.a.solr.common.cloud.ConnectionManager  : Waiting for client to connect to ZooKeeper

2016-07-24 13:29:01.953  INFO 83666 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn          : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

2016-07-24 13:29:01.955  INFO 83666 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn          : Socket connection established to localhost/127.0.0.1:2181, initiating session

2016-07-24 13:29:01.967  INFO 83666 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn          : Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1561cdd875e0004, negotiated timeout = 10000

2016-07-24 13:29:01.972  INFO 83666 --- [back-3-thread-1] o.a.solr.common.cloud.ConnectionManager  : Watcher org.apache.solr.common.cloud.ConnectionManager@4bb95d56 name:ZooKeeperConnection Watcher:localhost:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None

2016-07-24 13:29:01.972  INFO 83666 --- [qtp699221219-28] o.a.solr.common.cloud.ConnectionManager  : Client is connected to ZooKeeper

2016-07-24 13:29:01.973  INFO 83666 --- [qtp699221219-28] o.apache.solr.common.cloud.SolrZkClient  : Using default ZkACLProvider

2016-07-24 13:29:01.974  INFO 83666 --- [qtp699221219-28] o.a.solr.common.cloud.ZkStateReader      : Updating cluster state from ZooKeeper...



2016-07-24 13:29:01.990  INFO 83666 --- [qtp699221219-28] o.a.solr.common.cloud.ZkStateReader      : Loaded empty cluster properties

2016-07-24 13:29:01.995  INFO 83666 --- [qtp699221219-28] o.a.solr.common.cloud.ZkStateReader      : Updated live nodes from ZooKeeper... (0) -> (2)

2016-07-24 13:31:24.653 ERROR 83666 --- [qtp699221219-28] o.a.s.client.solrj.impl.CloudSolrClient  : Request to collection foo failed due to (0) java.net.ConnectException: Operation timed out, retry? 0

and my code is:

    String zkHostString = "localhost:2181";
    CloudSolrClient solr = new CloudSolrClient.Builder().withZkHost(zkHostString).build();
    solr.setDefaultCollection("foo");
    SolrQuery query = new SolrQuery();
    query.set("q", "*:*");
    QueryResponse response = null;
    try {
        response = solr.query(query);
    } catch (SolrServerException e) {
        return null;
    }
    //Do Something with the results...

Answer 1

Urgh, I'm an idiot, the zookeeper instance and solr instances are inside docker, the code posted above is not. So Zookeeper reported back the solr urls using the docker containers ip...The host needs to connect via localhost and not the docker container ip.

Eg: Zookeeper responds [ http://172.17.0.5:8983/solr/foo_shard1_replica2 , http://172.17.0.6:8984/solr/foo_shard1_replica1] but my code needs to call [ http://localhost:8983/solr/foo_shard1_replica2 , http://localhost:8984/solr/foo_shard1_replica1]

Answer 2

I think I have a similar problem. I have Solr+Zookeeper running on a remote machine (solr-1) on port 8983+9983 - internally they are a docker-compose network with 172.22.XX addresses (locally accessible for that machine, but not where my code is running). However the ports 8983 and 9983 are exposed and accessing the Solr Admin UI (running on solr-1:8983) works without problems.

When I try to connect using SolrJ, I am using the CloudSolrClient - initialised like this:

 new CloudSolrClient.Builder(zkHosts, Optional.empty()).build()

where zkHosts only contains ' solr-1:9983 " (escaped). As soon as I send a request, he calls getLiveNodes which returns the local address (172.22.XX) instead of the global hostname (solr-1:9983). Because this is obviously not accessible for me, I get a timeout.

Does this sound similar to your problem? Do you have any ideas? Do you think the issue is SolrJ or the way I setup the Solr/Zookeeper+Docker?

This is what my docker-compose file looks like:

version: '2'
services:
    solr:
        image: solr
        container_name: solr    
        ports:
            - "9983:9983"
            - "8983:8983"
        networks:
            - solr 
        entrypoint:
            - docker-entrypoint.sh
            - solr
            - start
            - -c
            - -f
networks:
    solr: