如何从 Rails 中的 elasticsearch 中检索所有记录

Question

There is an upper limit on the number of docs you can get from elastic search(that is 10000). we can use "scroll" to retrieve all the records. Does anyone know how to embed this in code?

There is this method scroll

https://github.com/elastic/elasticsearch-ruby/blob/4608fd144277941003de71a0cdc24bd39f17a012/elasticsearch-api/lib/elasticsearch/api/actions/scroll.rb

But I don't know how to use it. Could you explain how to use it?

I have tried the "scan". But it is no longer supported in Elasticsearch anymore.

# Open the "view" of the index
response = client.search index: 'test', search_type: 'scan', scroll: '5m', size: 10

# Call `scroll` until results are empty
while response = client.scroll(scroll_id: response['_scroll_id'], scroll: '5m') and not 
   response['hits']['hits'].empty? do
      puts response['hits']['hits'].map { |r| r['_source']['title'] }
end

Answer 1

Your code should work, but as you mentioned the scan parameter for search_type is not necessary. I just ran this locally with some test data and it worked:

# scroll.rb
require 'elasticsearch'

client = Elasticsearch::Client.new

response = client.search(index: 'articles', scroll: '10m')
scroll_id = response['_scroll_id']
while response['hits']['hits'].size.positive?
  response = client.scroll(scroll: '5m', body: { scroll_id: scroll_id })
  puts(response['hits']['hits'].map { |r| r['_source']['title'] })
end

Output:

$ ruby scroll.rb                                                                                         
Title 297                                                                                                
Title 298                                                                                                
Title 299                                                                                                
Title 300
...

You can fiddle around with the value for the scroll parameter, but something like this should work for you too.

Answer 2

paragraph from elastic official docs:

We no longer recommend using the scroll API for deep pagination. If you need to preserve the index state while paging through more than 10,000 hits, use the search_after parameter with a point in time (PIT).

Scroll Official Doc Link

I recommend to use pagination.

you can use

that limitation in number of hits is for performance inprovements, you can use pagination, Its much faster.

in this way you can use start point with form key or use search_after key with sort and PIT(point in time for prevent from inconsistent result). and you can determinate you hits size key with 10 for faster query time.

Pagination Official Doc Link

for instantiate PIT ID:

POST /test/_pit?keep_alive=1m

for instantiate pagination:

GET /test/_search
{
  "size": 10,
  "query": {
    "match" : {
      "user.id" : "elkbee"
    }
  },
  "pit": {
    "id":  "46ToAwMDaWR5BXV1aWQyKwZub2RlXzMAAAAAAAAAACoBYwADaWR4BXV1aWQxAgZub2RlXzEAAAAAAAAAAAEBYQADaWR5BXV1aWQyKgZub2RlXzIAAAAAAAAAAAwBYgACBXV1aWQyAAAFdXVpZDEAAQltYXRjaF9hbGw_gAAAAA==", 
    "keep_alive": "1m"
  },
  "sort": [ 
    {"@timestamp": {"order": "asc", "format": "strict_date_optional_time_nanos", "numeric_type" : "date_nanos" }}
  ]
}

for get rest of data in pagination: there is sort key in the result, put it in the search_after

GET /test/_search
{
  "size": 10,
  "query": {
    "match" : {
      "user.id" : "elkbee"
    }
  },
  "pit": {
    "id":  "46ToAwMDaWR5BXV1aWQyKwZub2RlXzMAAAAAAAAAACoBYwADaWR4BXV1aWQxAgZub2RlXzEAAAAAAAAAAAEBYQADaWR5BXV1aWQyKgZub2RlXzIAAAAAAAAAAAwBYgACBXV1aWQyAAAFdXVpZDEAAQltYXRjaF9hbGw_gAAAAA==", 
    "keep_alive": "1m"
  },
  "sort": [
    {"@timestamp": {"order": "asc", "format": "strict_date_optional_time_nanos"}}
  ],
  "search_after": [                                
    "2021-05-20T05:30:04.832Z", #you can find this value from sort key in response 
    4294967298
  ],
  "track_total_hits": false                        
}