Fetch data from elasticsearch in pieces using Node js

Question

I am trying to query a large amount of data from elastic using node js. I have the following query which works perfectly -

client.search({
  "index": 'test',
  "size": 50,
  "type": consts.ReportType.VIOLATION,
  "body": searchQuery
}, callback);

My main goal is to fetch data in pieces, each time to get only 50 results as I'll probably have thousands of docs in elastic. I'm displaying the results in pages at client side (let's say like google search results) and wish to get more data from elastic only if needed.

So, is it possible to maintain some index which will tell elastic which is the last response I already got, and to fetch another 50 results from that point?

Thanks

Answer 1

You can use pagination to perform this.

So, in your case, for example :

client.search({
  "index": 'test',
  "from": 2,
  "size": 50,
  "type": consts.ReportType.VIOLATION,
  "body": searchQuery
}, callback);

Answer 2

Change the from size for next 50 results

GET /_search { "from" : 0, "size" : 50, "query" : { "term" : { "user" : "kimchy" } } }

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-from-size.html

Get bulk data Check scroll api https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/api-reference.html#api-scroll

Answer 3

更好的方法是使用 Elastic Scroll API