在Solr组件准备方法中执行分布式搜索

Question

I'm writing a custom Solr component. In the component's prepare method I'm executing a query given as a custom parameter (inside the req.params ). I'm not running the q parameter query in the prepare method, but another input query defined in a custom parameter. I'm using the documents returned by that custom input query to do some preparations in the prepare method.

The problem is that since my index is distributed into several shards, the documents returned by the custom query are only the ones residing on one of the shards. In other words, the search performed in my prepare method is not distributed, and I'm getting partial results. This is more or less how I perform the search in my prepare method:

rb.req.getSearcher().getDocList(customQuery, null, null, offset, len, 0);

Is there a way to make a distributed search in the prepare method and get the matched documents from all the shards?

---

EDIT:

My current solution is to execute a query using Solrj roughly as follows:

SolrServer server = new HttpSolrServer(url);
SolrQuery request = new SolrQuery(customQuery);
NamedList queryResponse = server.query(request).getResponse();

Then I parse the response to get the content of the returned documents. I don't like my solution for several reasons. One of the reasons is that I have to parse the response. But the main reason is that I have to pass the Solr server url as a parameter. I put the url in the solrconfig.xml file. Is it possible to somehow construct a SolrServer instance without explicitly stating the Solr server url (perhaps through ZooKeeper)?

Answer 1

The Easy Way

Use CloudSolrServer to execute the distributed query. Feed it the Zookeeper url and the collection name (which are available in the response builder):

CoreDescriptor coreDescriptor = rb.req.getCore().getCoreDescriptor();
String collectionName = coreDescriptor.getCloudDescriptor().getCollectionName();    
ZkController zkController = coreDescriptor.getCoreContainer().getZkController();    
String zookeeperUrl = zkController.getZkServerAddress();

CloudSolrServer server = new CloudSolrServer(zookeeperUrl);
server.setDefaultCollection(collectionName);
server.connect();

SolrRequest request = ... //initialize the solr request to execute the query
NamedList<Object> solrResponse = server.request(solrRequest);
// do whatever you like with the returned response;
server.shutdown();

---

The Right Way

Do not perform a distributed search inside the prepare method. Don't query the index in the prepare method. What you have to do is first decide at which stage of the execution you want your distributed query to be executed. The stages are STAGE_START , STAGE_PARSE_QUERY , STAGE_TOP_GROUPS , STAGE_EXECUTE_QUERY , STAGE_GET_FIELDS and STAGE_DONE . If you need it to be executed between two of the stages, then create a new intermediate stage (such as EXECUTE_PREPARING_QUERY ).

Override the distributedProcess method and implement it in such a way that if the current stage is your stage then set the right parameters for the shard request:

@Override public int distributedProcess(ResponseBuilder rb) {
    ...
    if (rb.stage == MY_STAGE) {
       ShardRequest sreq = new ShardRequest();
       sreq.purpose = ShardRequest.PURPOSE_PRIVATE;
       sreq.params = new ModifiableSolrParams();
       // set the parameters for the shard request
       rb.addRequest(this, sreq);
    }
    ...
}

Now each shard is going to execute the request defined by the params you've set on its own core. That's going to happen on the stage MY_STAGE . You still have to handle the responses of the shards, combine them and use them. The right place to handle all those responses is inside the handleResponses method of the component. So override handleResponses and do whatever you need to do with the shard responses if you're in the right stage. You probably need to save them somewhere so you can reference them later in the finishStage method.

@Override public void handleResponses(ResponseBuilder rb, ShardRequest sreq) {
   ...
   if (stage == MY_STAGE) {
      List<ShardResponse> responses = sreq.responses;
      for (ShardResponse response : responses) {
         //do something with the response, maybe save it somewhere
         rb.finished.remove(sreq);
      }
   }
   ...
}

Now you have to override the finishStage method and do whatever you need to do with the combined results.

@Override public void finishStage(ResponseBuilder rb) {
   ...
   if (rb.stage == MY_STAGE) {
      // do whatever you need to do with the results
   }
   ...
}

The important message is to use the response builder stages to control the execution flow of the component with relation to the other components. You don't have to put the code in the prepare method if you want it to be executed before the execution of the actual query. You just have to create or use a stage that's intermediate to STAGE_START and STAGE_EXECUTE_QUERY .