比較elasticsearch索引中不同的文檔字段值

Question

只是想知道如何比較 2 個不同文檔的值。 我正在索引中攝取以下值。

我希望查詢和比較每個 plugin_instance 的“allocated-mb”>“max-mb”的每個文檔字段“type_instance”值。

POST test/_bulk
{"index":{"_id":1}}
{"plugin_instance": "root-yarn-queue-name-1", "type_instance": "allocated-mb", "value": 4024}
{"index":{"_id":2}}
{ "plugin_instance": "root-yarn-queue-name-1", "type_instance": "max-mb", "value": 2048}
{"index":{"_id":3}}
{"plugin_instance": "root-yarn-queue-name-2", "type_instance": "max-mb", "value": 3048}
{"index":{"_id":4}}
{"root-yarn-queue-name-2", "type_instance": "allocated-mb", "value": 1028}
{"index":{"_id":5}}
{"plugin_instance": "some-random-queue-name-2", "type_instance": "allocated-mb", "value": 2028}
{"index":{"_id":6}}
{"plugin_instance": "some-random-queue-name-2", "type_instance": "max-mb", "value": 2028}

---

只是想知道實現以下目標的簡單方法是什么

1. 選擇帶有 plugin_instance=root-yarn-queue-name-* 的記錄
2. 選擇帶有 type_instance 的記錄（allocated-mb、located-vcores、max-mb、max-vcores）
3. 將具有相同 plugin_instance 的記錄分組（獲取存儲桶中隊列的記錄）
4. 比較分配的 mb > max-mb 和分配的 vcore > max-vcore 的值，並選擇在給定時間段內滿足所有這些條件的那些記錄

到目前為止，我已經設法根據 plugin_instance 對文檔進行存儲。 想知道根據“type_instance”比較每個存儲桶的文檔的簡單方法是什么

GET test/_search
{
  "query": {
    "query_string": {
      "fields": [
        "plugin_instance.keyword",
        "type_instance.keyword"
        ],
      "query": "root-yarn-queue-name-* AND (max-mb OR allocated-mb)"
    }
  },
    "aggs": {
    "byField": {
      "terms": {
        "field": "plugin_instance.keyword"
      }
    }
  }
}

Answer 1

解決上述問題的一種方法是在客戶端獲取文檔和執行過濾。 第二個將涉及使用bucket_selector 聚合

映射

{
  "index60" : {
    "mappings" : {
      "properties" : {
        "plugin_instance" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "type_instance" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "value" : {
          "type" : "long"
        }
      }
    }
  }
}

數據：

  "hits" : [
      {
        "_index" : "index60",
        "_type" : "_doc",
        "_id" : "vCN--3AB_Wo5Rvhl5c7f",
        "_score" : 1.0,
        "_source" : {
          "plugin_instance" : "root-yarn-queue-name-1",
          "type_instance" : "max-vcore",
          "value" : 1000
        }
      },
      {
        "_index" : "index60",
        "_type" : "_doc",
        "_id" : "vSN--3AB_Wo5Rvhl9M6R",
        "_score" : 1.0,
        "_source" : {
          "plugin_instance" : "root-yarn-queue-name-1",
          "type_instance" : "max-mb",
          "value" : 2048
        }
      },
      {
        "_index" : "index60",
        "_type" : "_doc",
        "_id" : "viN--3AB_Wo5Rvhl_s5m",
        "_score" : 1.0,
        "_source" : {
          "plugin_instance" : "root-yarn-queue-name-2",
          "type_instance" : "max-mb",
          "value" : 3048
        }
      },
      {
        "_index" : "index60",
        "_type" : "_doc",
        "_id" : "wCN_-3AB_Wo5Rvhlhc53",
        "_score" : 1.0,
        "_source" : {
          "plugin_instance" : "root-yarn-queue-name-2",
          "type_instance" : "allocated-mb",
          "value" : 1028
        }
      },
      {
        "_index" : "index60",
        "_type" : "_doc",
        "_id" : "wSOA-3AB_Wo5RvhlCc5r",
        "_score" : 1.0,
        "_source" : {
          "plugin_instance" : "root-yarn-queue-name-1",
          "type_instance" : "allocated-mb",
          "value" : 3000
        }
      }
    ]

詢問：

1. 創建一個桶 root-yarn-queue-name-.*
2. 創建 ["max-mb","allocated-mb"] 的子桶
3. 找到 ["max-mb","allocated-mb"] 的最大值
4. 查找分配的 mb 的值
5. 選擇一個已分配-mb 具有最大值的存儲桶

{
  "size": 0,
  "aggs": {
    "plugin_instance": {
      "terms": {
        "field": "plugin_instance.keyword",
        "include": "root-yarn-queue-name-.*", --> pattern
        "size": 10000
      },
      "aggs": {
        "instance": {
          "terms": {
            "field": "type_instance.keyword",
            "include": [  --> create a bucket of these two
              "max-mb",
              "allocated-mb"
            ],
            "size": 10
          }
        },
        "maxValue": {
          "max": {
            "field": "value"
          }
        },
        "allocated-mb": {
          "filter": {
            "term": {
              "type_instance.keyword": "allocated-mb"
            }
          },
          "aggs": {
            "filtered_maxValue": {
              "max": {
                "field": "value"
              }
            }
          }
        },
        "my_bucket": {
          "bucket_selector": {
            "buckets_path": {
              "filteredValue": "allocated-mb>filtered_maxValue",
              "maxValue": "maxValue"
            },
            "script": "params.filteredValue==params.maxValue"
          }
        }
      }
    }
  }
}

結果：

 "aggregations" : {
    "plugin_instance" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "root-yarn-queue-name-1",
          "doc_count" : 3,
          "instance" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : "allocated-mb",
                "doc_count" : 1
              },
              {
                "key" : "max-mb",
                "doc_count" : 1
              }
            ]
          },
          "maxValue" : {
            "value" : 3000.0
          },
          "allocated-mb" : {
            "doc_count" : 1,
            "filtered_maxValue" : {
              "value" : 3000.0
            }
          }
        }
      ]
    }
  }

如果您對此有任何疑問，請告訴我