Comparing the different document field values in elasticsearch index

Question

Just wondering How can I compare the value from 2 different documents . I am ingesting the following values in the index.

I am looking to query and compare of each document field "type_instance" value of "allocated-mb" > "max-mb" of the each plugin_instance .

POST test/_bulk
{"index":{"_id":1}}
{"plugin_instance": "root-yarn-queue-name-1", "type_instance": "allocated-mb", "value": 4024}
{"index":{"_id":2}}
{ "plugin_instance": "root-yarn-queue-name-1", "type_instance": "max-mb", "value": 2048}
{"index":{"_id":3}}
{"plugin_instance": "root-yarn-queue-name-2", "type_instance": "max-mb", "value": 3048}
{"index":{"_id":4}}
{"root-yarn-queue-name-2", "type_instance": "allocated-mb", "value": 1028}
{"index":{"_id":5}}
{"plugin_instance": "some-random-queue-name-2", "type_instance": "allocated-mb", "value": 2028}
{"index":{"_id":6}}
{"plugin_instance": "some-random-queue-name-2", "type_instance": "max-mb", "value": 2028}

---

just wonder what would the easy way to achieve following

1. Select records with plugin_instance=root-yarn-queue-name-*
2. select records with type_instance in (allocated-mb, allocated-vcores, max-mb, max-vcores)
3. group records with same plugin_instance (to get records for a queue in a bucket)
4. compare if value of allocated-mb > max-mb and allocated-vcore > max-vcore and select those records which fulfil all these conditions in a given time period

till now I have managed to bucket the document based on the plugin_instance . Wondering what would be the easy way to compare the documents of each bucket based on the "type_instance"

GET test/_search
{
  "query": {
    "query_string": {
      "fields": [
        "plugin_instance.keyword",
        "type_instance.keyword"
        ],
      "query": "root-yarn-queue-name-* AND (max-mb OR allocated-mb)"
    }
  },
    "aggs": {
    "byField": {
      "terms": {
        "field": "plugin_instance.keyword"
      }
    }
  }
}

Answer 1

One way to solve above will be to get documents and peform filtering at client side. Second will involve use of bucket_selector aggregation

Mapping

{
  "index60" : {
    "mappings" : {
      "properties" : {
        "plugin_instance" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "type_instance" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "value" : {
          "type" : "long"
        }
      }
    }
  }
}

Data:

  "hits" : [
      {
        "_index" : "index60",
        "_type" : "_doc",
        "_id" : "vCN--3AB_Wo5Rvhl5c7f",
        "_score" : 1.0,
        "_source" : {
          "plugin_instance" : "root-yarn-queue-name-1",
          "type_instance" : "max-vcore",
          "value" : 1000
        }
      },
      {
        "_index" : "index60",
        "_type" : "_doc",
        "_id" : "vSN--3AB_Wo5Rvhl9M6R",
        "_score" : 1.0,
        "_source" : {
          "plugin_instance" : "root-yarn-queue-name-1",
          "type_instance" : "max-mb",
          "value" : 2048
        }
      },
      {
        "_index" : "index60",
        "_type" : "_doc",
        "_id" : "viN--3AB_Wo5Rvhl_s5m",
        "_score" : 1.0,
        "_source" : {
          "plugin_instance" : "root-yarn-queue-name-2",
          "type_instance" : "max-mb",
          "value" : 3048
        }
      },
      {
        "_index" : "index60",
        "_type" : "_doc",
        "_id" : "wCN_-3AB_Wo5Rvhlhc53",
        "_score" : 1.0,
        "_source" : {
          "plugin_instance" : "root-yarn-queue-name-2",
          "type_instance" : "allocated-mb",
          "value" : 1028
        }
      },
      {
        "_index" : "index60",
        "_type" : "_doc",
        "_id" : "wSOA-3AB_Wo5RvhlCc5r",
        "_score" : 1.0,
        "_source" : {
          "plugin_instance" : "root-yarn-queue-name-1",
          "type_instance" : "allocated-mb",
          "value" : 3000
        }
      }
    ]

Query:

1. Create a bucket of root-yarn-queue-name-.*
2. Create sub buckets of ["max-mb","allocated-mb"]
3. Find max value of ["max-mb","allocated-mb"]
4. Find value of allocated-mb
5. Select a bucket where allocated-mb has max value

{
  "size": 0,
  "aggs": {
    "plugin_instance": {
      "terms": {
        "field": "plugin_instance.keyword",
        "include": "root-yarn-queue-name-.*", --> pattern
        "size": 10000
      },
      "aggs": {
        "instance": {
          "terms": {
            "field": "type_instance.keyword",
            "include": [  --> create a bucket of these two
              "max-mb",
              "allocated-mb"
            ],
            "size": 10
          }
        },
        "maxValue": {
          "max": {
            "field": "value"
          }
        },
        "allocated-mb": {
          "filter": {
            "term": {
              "type_instance.keyword": "allocated-mb"
            }
          },
          "aggs": {
            "filtered_maxValue": {
              "max": {
                "field": "value"
              }
            }
          }
        },
        "my_bucket": {
          "bucket_selector": {
            "buckets_path": {
              "filteredValue": "allocated-mb>filtered_maxValue",
              "maxValue": "maxValue"
            },
            "script": "params.filteredValue==params.maxValue"
          }
        }
      }
    }
  }
}

Result:

 "aggregations" : {
    "plugin_instance" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "root-yarn-queue-name-1",
          "doc_count" : 3,
          "instance" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : "allocated-mb",
                "doc_count" : 1
              },
              {
                "key" : "max-mb",
                "doc_count" : 1
              }
            ]
          },
          "maxValue" : {
            "value" : 3000.0
          },
          "allocated-mb" : {
            "doc_count" : 1,
            "filtered_maxValue" : {
              "value" : 3000.0
            }
          }
        }
      ]
    }
  }

Let me know if you have any doubt in this