Elasticsearch Scan&scroll with JEST API

Question

I am currently working with JEST: https://github.com/searchbox-io/Jest

Is it possible to do scan&scroll with this API?

http://www.elasticsearch.org/guide/reference/api/search/search-type/

I am currently using the Search command:

Search search = new Search("{\"size\" : "+RESULT_SIZE+", \"query\":{\"match_all\":{}}}");

but am worried about large result sets. If you use the Search command for this how do you set the "search_type=scan&scroll=10m&size=50" arguments?

Answer 1

Is it possible to do scan&scroll with this API?

Yes it is. My implementation it's working like this.

Start the scroll search on elastic search:

    public SearchResult startScrollSearch (String type, Long size) throws IOException {

            String query = ConfigurationFactory.loadElasticScript("my_es_search_script.json");

            Search search = new Search.Builder(query)
                                            // multiple index or types can be added.
                                            .addIndex("myIndex")
                                            .addType(type)
                                            .setParameter(Parameters.SIZE, size)
                                            .setParameter(Parameters.SCROLL, "1m")
                                            .build();

                SearchResult searchResult = EsClientConn.getJestClient().execute(search);
                return searchResult;

        }

SearchResult object will return the first (size) itens off the search as usual but will return to a scrollId parameter that is a reference to remain resultSet that elasticSearch keeps in memory for you. Parameters.SCROLL, will define the time that this search will be keeped on memory.

For read the scrollId:

scrollId = searchResult.getJsonObject().get("_scroll_id").getAsString();

For read more items from the resultSet you should use something like follow:

public JestResult readMoreFromSearch(String scrollId, Long size) throws IOException {

    SearchScroll scroll = new SearchScroll.Builder(scrollId, "1m")
                .setParameter(Parameters.SIZE, size).build();

        JestResult searchResult = EsClientConn.getJestClient().execute(scroll);
        return searchResult;

}

Don't forget that each time you read from the result set a new scrollId is returned from elastic.

Please tell me if you have any doubt.

Answer 2

Agreed we need to catch up however please open an issue if you need a feature.

Please check https://github.com/searchbox-io/Jest/blob/master/jest/src/test/java/io/searchbox/core/SearchScrollIntegrationTest.java at master

Answer 3

EDIT:

It doesn't appear that JEST currently supports the "Scan" search type: In a wicked fast turnaround, it appears that JEST now supports Scan type searches! Props to @Ferhat for the quick turnaround! JEST - SearchType.java

---

Have you considered just using the ElasticSearch Transport client? I could understand if you like the JEST API a little better, but as new features roll out for ElasticSearch ( Exhibit A: ElasticSearch 0.90 is fantastic! ), you'll get to have them as soon as they pop out instead of waiting for JEST to catch up.

My $0.02.