Elasticsearch aggregation over children document field values

Question

I'm facing the following problem of selecting and sorting parent documents based on an aggregated value over its children documents. The aggregation (eg sum) itself depends on a query string, ie which children documents are relevant for the aggregation.

Example: Given the documents basket A and basket B , for each basket document , I am looking to sum over the number field of its fruit children if the name field matches my query, eg apples .

PUT /baskets/_doc/0
{
  "name": "basket A", 
  "fruit": [
    {
      "name": "apples",
      "number": 2
    },
    {
      "name": "oranges",
      "number": 3
    }
  ]
}

PUT /baskets/_doc/1
{
  "name": "basket B",
  "fruit": [
    {
      "name": "apples",
      "number": 3
    },
    {
      "name": "apples",
      "number": 3
    }
  ]
}

Mappings:

PUT /baskets
{
  "mappings": {
    "properties": {
      "name": { "type": "text" },
      "fruit": { 
        "type": "nested",
        "properties": {
          "name": { "type": "text" },
          "number": { "type": "long" }
        }
      }
    }
  }
}

• Use case 1: Which basket has (strictly) more than 5 apples? Would expect only basket B
• Use case 2: Sort baskets by number of apples. Would expect basket B with a total of 6 apples, then basket A with a total of 2 apples.

How can one implement this using the Elasticsearch (7.8.0) query DSL?

I have tried so far with nested queries and aggregations without success.

Thanks!

Edit: Added mappings

Edit: Updated the numbers to better reflect the problem

*Edit: Added possible answer to Use case 2 (see comments to the answer from @joe):

GET /profiles/_search
{
  "aggs": {
    "aggs_baskets": {
      "terms": {
        "field": "name",
        "order": {"nest > fruit_filter > fruit_sum": "desc"}
      },
      "aggs": {
        "nest":{
          "nested":{
            "path": "fruit"
          },
          "aggs":{
            "fruit_filter":{
              "filter": {
                "term": {"fruit.name": "apple"}
              },
              "aggs":{
                "fruit_sum":{
                  "sum": {"field": "fruit.number"}
                }
              }
            }
          }
        }
      }
    }
  }
}

Answer 1

Use case 1:

GET baskets/_search
{
  "query": {
    "nested": {
      "path": "fruit",
      "inner_hits": {}, 
      "query": {
        "bool": {
          "must": [
            {
              "term": {
                "fruit.name": {
                  "value": "apples"
                }
              }
            },
            {
              "range": {
                "fruit.number": {
                  "gte": 5
                }
              }
            }
          ]
        }
      }
    }
  }
}

Strictly more than 5 --> gt ; >=5 --> gte .

Also notice the inner_hits part -- this gives you the actual nested subdocument which caused this particular basket to match the query. It's not required but good-to-know.

Use case 2:

GET baskets/_search
{
  "sort": [
    {
      "fruit.number": {
        "nested_path": "fruit",
        "order": "desc"
      }
    }
  ]
}

Use case 2 Edit:

There are probably cleaner ways of doing this but I'd go with the following:

GET baskets/_search
{
  "size": 0,
  "aggs": {
    "multiply_and_add": {
      "scripted_metric": {
        "params": {
          "only_fruit_name": "apples"
        },
        "init_script": "state.by_basket_name = [:]",
        "map_script": """
          def basket_name = params._source['name'];
          def fruits = params._source['fruit'].findAll(group -> group.name == params.only_fruit_name);
          
          for (def fruit_group : fruits) {
            def number = fruit_group.number;
            
            if (state.by_basket_name.containsKey(basket_name)) {
              state.by_basket_name[basket_name] += number;
            } else {
              state.by_basket_name[basket_name] = number;
            }
          }
        """,
        "combine_script": "return state.by_basket_name",
        "reduce_script": "return states"
      }
    }
  }
}

yielding a hash map along the lines of

{
  ...
  "aggregations":{
    "multiply_and_add":{
      "value":[
        {
          "basket A":2,
          "basket B":6
        }
      ]
    }
  }
}

Sorting can either be done in the reduce_script or within your ES response post-processing pipeline. You could of course choose to go w/ (sorted) lists and lambdas ...

Notice the required nested_path .

Answer 2

After a while of searching and testing, here are (in addition to @joe's answer to use case 2 ) possible queries for both use cases. Note that both use cases require to change the mapping for the field name to be of type keyword .

Use case 1 : Which basket has (strictly) more than 5 apples? Would expect only basket B

For more information on filtering results by their aggregation value see Bucket Selectors

GET /baskets/_search
{
  "aggs": {
    "aggs_baskets": {
      "terms": {
        "field": "name"
      },
      "aggs": {
        "nest":{
          "nested":{
            "path": "fruit"
          },
          "aggs":{
            "fruit_filter":{
              "filter": {
                "match": {"fruit.name": "apples"}
              },
              "aggs":{
                "fruit_sum":{
                  "sum": {"field": "fruit.number"}
                }
              }
            }
          }
        },
        "basket_sum_filter":{
          "bucket_selector":{
            "buckets_path":{
              "fruitSum":"nest > fruit_filter > fruit_sum"
            },
            "script":"params.fruitSum > 5"
          }
        }
      }
    }
  }
}

... yielding

...,

"buckets": [
    {
        "key": "basket B",
        "doc_count": 1,
        "nest": {
            "doc_count": 2,
            "fruit_filter": {
                "doc_count": 2,
                "fruit_sum": {
                    "value": 6
                }
            }
        }
    }
]

Use case 2 : Sort baskets by number of apples. Would expect basket B with a total of 6 apples, then basket A with a total of 2 apples.

GET /baskets/_search
{
  "aggs": {
    "aggs_baskets": {
      "terms": {
        "field": "name",
        "order": {"nest > fruit_filter > fruit_sum": "desc"}
      },
      "aggs": {
        "nest":{
          "nested":{
            "path": "fruit"
          },
          "aggs":{
            "fruit_filter":{
              "filter": {
                "term": {"fruit.name": "apple"}
              },
              "aggs":{
                "fruit_sum":{
                  "sum": {"field": "fruit.number"}
                }
              }
            }
          }
        }
      }
    }
  }
}

... yielding

...,

"buckets": [
    {
        "key": "basket B",
        "doc_count": 1,
        "nest": {
            "doc_count": 2,
            "fruit_filter": {
                "doc_count": 2,
                "fruit_sum": {
                    "value": 6
                }
            }
        }
    },
    {
        "key": "basket A",
        "doc_count": 1,
        "nest": {
            "doc_count": 2,
            "fruit_filter": {
                "doc_count": 1,
                "fruit_sum": {
                    "value": 2
                }
            }
        }
    }
]