如何用elasticsearch进行分页？来自vs scroll API

Question

I'm using elasticsearch as DB to store a large batch of log data. I know there are 2 ways to do pagination:

1. Use size and from API

2. Use scroll API

Now I'm using 'from' to do pagination.Get page and size parameters from front end,and at back end(Java)

searchSourceBuilder.size(size);
searchSourceBuilder.from(page * size);

However, if page*size > 10000, an exception thrown from ES.

Can I use scroll API to do pagination?

I know that if I use scroll API, the searchResponse object will return me a _scroll_id , which looks like a base64 string.

How can I control page and size?

It seems Scroll API only support successive page number?

Answer 1

There is nothing in Elasticsearch which allows direct jump to a specific page as the results have to be collected from different shards. So in your case search_after will be a better option. You can reduce the amount of data returned for the subsequent queries and then once you reach the page which is actually requested get the complete data.

Example: Let's say you have to jump to 99th page then you can reduce the amount of data for all 98th pages request and once you're at 99 you can get the complete data.

Answer 2

What you said is correct!

You can't do traditional pagination by using scroll API.

I might suggest you to look the Search After API

This might not help you to cover your requirement!

The From / Size default max result size is 10,000.

As it is mentioned here

Note that from + size can not be more than the index.max_result_window index setting which defaults to 10,000

So if you somehow increase the index.max_result_window setting persistent then it will increase the max number of search result! But this might not be the solution but decrease the limitation.

Remember this above solution might hamper the performance of your ES server. Read through all the posts here.

My suggestion is to use Scroll API and change the pagination style