如何在不担心客户端Java堆内存的情况下使用Java Client使用EasticSearch进行查询

Question

I am new to elastic search, I was reading that we can use elasticsearch to query using its rest API calls.

I was reading the following link : http://blogs.justenougharchitecture.com/using-jest-as-a-rest-based-java-client-with-elasticsearch/

Is this the right way to do it??

Also, I donot want to put a limit to the number of results that my search will return(it can return millions of records).

So just how ResultSet in java works, where the table might have millions of row, but we can iterate one row at a time and just process it, and not storing it in my java heap anywhere), hence not worrying about the java heap space,.. Similarly I want to do something similar with Elastic Search Querying if possible, ( where I want all the records in the query), but not putting them all together in my memory while iterating them.

Is it possible to do so using any java client(via rest API), if not via rest API, then is there a method of solving this problem.

Thanks

Answer 1

First, if you use a Java or another JVM language, you could also use the native client . Jest is a good option if you want to keep your dependencies small (the java client is essentially the same as the complete server) or if you want or can access Elasticsearch only via the HTTP interface and not via its binary interface.

Second, what you want to use is the scroll API: https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-search-scrolling.html (didn't found a quick reference on the Jest documentation though). It doesn't exactly work like ResultSet, but allows you to iterate in chunks over all your results. An example, copied from the documentation

QueryBuilder query = ...;

SearchResponse scrollResponse = client.prepareSearch(index)
        .setSearchType(SearchType.SCAN)
        .setScroll(new TimeValue(60000)) // timeout
        .setQuery(query)
        .setSize(100) // bulk size
        .execute().actionGet(); 

//Scroll until no hits are returned
while (!scrollResp.getHits().getHits().isEmpty()) {

    for (SearchHit hit : scrollResp.getHits().getHits()) {
        //Handle the hit...
    }

    scrollResp = client.prepareSearchScroll(scrollResp.getScrollId())
        .setScroll(new TimeValue(60000))
        .execute().actionGet();
}