How to scroll Data using Scroll API elasticsearch

Question

i am new to elk stack

• i have tried from this but not getting working flow ..

• for example executed below search query

POST <index-name>/_search?scroll=2m
{
  "query": {"match_all": {}}
}

• and got the scroll_id from this query then tried Retrieving the next batch of results for a scrolling search.using this

GET /_search/scroll
{
  "scroll_id" : "<scroll_id>"
}

• got result first time

"took" : 2,
  "timed_out" : false,
  "terminated_early" : true,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 13059,
      "relation" : "eq"
    }

• my question is why i am getting error when i tried scrolling again using same scroll_id

"caused_by" : {
      "type" : "search_context_missing_exception",
      "reason" : "No search context found for id"

• Versions Used

Kibana 7.9.3
Elastic Search 7.9.3

Answer 1

The scroll_id value changes in every response. So the next search call needs to use the new scroll id from the previous search response.

You started correctly with

POST <index-name>/_search?scroll=2m
{
  "query": {"match_all": {}}
}

In the response you get, a field called _scroll_id contains the next scroll id to use for the next call (like a cursor), let's call it scroll_id_1 :

GET /_search/scroll
{
  "scroll_id" : "<scroll_id_1>",
  "scroll": "2m"
}

In that next response, you get a new _scroll_id value (let's call it scroll_id_2 ) that you need to use it for the next call:

GET /_search/scroll
{
  "scroll_id" : "<scroll_id_2>",
  "scroll": "2m"
}

And you keep doing it until you get an empty result set, at which point you can clear the search context

DELETE /_search/scroll
{
  "scroll_id" : "<scroll_id_n>"
}