如何在 Elasticsearch 中对术语聚合结果进行分页

Question

I've been trying to figure out a way to paginate the results of a terms aggregation in Elasticsearch and so far I have not been able to achieve the desired result.

Here's the problem I am trying to solve. In my index, I have a bunch of documents that have a score (separate to the ES _score) that is calculated based on the values of the other fields in the document. Each document "belongs" to a customer, referenced by the customer_id field. The document also has an id, referenced by the doc_id field, and is the same as the ES meta-field _id . Here is an example.

{
 '_id': '1',
 'doc_id': '1',
 'doc_score': '85',
 'customer_id': '123'
}

For each customer_id there are multiple documents, all with different document ids and different scores. What I want to be able to do is, given a list of customer ids, return the top document for each customer_id (only 1 per customer) and be able to paginate those results similar to the size , from method in the regular ES search API. The field that I want to use for the document score is the doc_score field.

So far in my current Python script, I've tried is a nested aggs with a "top hits" aggregation to only get the top document for each customer.

{
 "size": 0,
 "query:": {
  "bool": {
   "must": [
    {
     "match_all": {}
    },
    {
     "terms": {
      "customer_id": customer_ids # a list of the customer ids I want documents for
     }
    },
    {
     "exists": {
      "field": "score" # sometimes it's possible a document does not have a score
     }
    }
   ]
  }
 }
 "aggs": {
  "customers": {
   "terms" : {
    {"field": "customer_id", "min_doc_count": 1},
    "aggs": {
     "top_documents": {
      "top_hits": {
       "sort": [
        {"score": {"order": "desc"}}
       ],
       "size": 1
      }
     }
    }
   }
  }
 }
}

I then "paginate" by going through each customer bucket, appending the top document blob to a list and then sorting the list based on the value of the score field and finally taking a slice documents_list[from:from+size] .

The issue with this is that, say I have 500 customers in the list but I only want the 2nd 20 documents, ie size = 20 , from=20 . So each time I call the function I have to first get the list for each of the 500 customers and then slice. This sounds very inefficient and is also a speed issue, since I need that function to be as fast as I can possibly make it.

Ideally, I could just get the 2nd 20 directly from ES without having to do any slicing in my function.

I have looked into Composite aggregations that ES offers, but it looks to me like I would not be able to use it in my case, since I need to get the entire doc, ie everything in the _source field in the regular search API response.

I would greatly appreciate any suggestions.

Answer 1

The best way to do this would be to use partitions

According to documentation:

GET /_search
{
   "size": 0,
   "aggs": {
      "expired_sessions": {
         "terms": {
            "field": "account_id",
            "include": {
               "partition": 1,
               "num_partitions": 25
            },
            "size": 20,
            "order": {
               "last_access": "asc"
            }
         },
         "aggs": {
            "last_access": {
               "max": {
                  "field": "access_date"
               }
            }
         }
      }
   }
}

https://www.elastic.co/guide/en/elasticsearch/reference/6.8/search-aggregations-bucket-terms-aggregation.html#_filtering_values_with_partitions