如何通過curl查詢Logstash並僅返回特定字段

Question

現在，我正在使用“ match_all”查詢來獲取Logstash正在處理的數據。 我得到的輸出是應該作為事件一部分的每個字段。 這是我的查詢：

{
"query": {
    "match_all" : { }
},
  "size": 1,
  "sort": [
{
 "@timestamp": {
     "order": "desc"
  }
  }
  ]
}

如您所見，我還在對我的結果進行排序，以便始終獲得最新輸出的結果。

這是我的輸出示例：

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 15768,
    "max_score" : null,
    "hits" : [
      {
        "_index" : "filebeat-2017.02.24",
        "_type" : "bro",
        "_id" : "AVpx-pFtiEtl3Zqhg8tF",
        "_score" : null,
        "_source" : {
          "resp_pkts" : 0,
          "source" : "/usr/local/bro/logs/current/conn.log",
          "type" : "bro",
          "id_orig_p" : 56058,
          "duration" : 848.388112,
          "local_resp" : true,
          "uid" : "CPndOf4NNf9CzTILFi",
          "id_orig_h" : "192.168.137.130",
          "conn_state" : "OTH",
          "@version" : "1",
          "beat" : {
            "hostname" : "localhost.localdomain",
            "name" : "localhost.localdomain",
            "version" : "5.2.0"
          },
          "host" : "localhost.localdomain",
          "id_resp_h" : "192.168.137.141",
          "id_resp_p" : 22,
          "resp_ip_bytes" : 0,
          "offset" : 115612,
          "orig_bytes" : 32052,
          "local_orig" : true,
          "input_type" : "log",
          "orig_ip_bytes" : 102980,
          "orig_pkts" : 1364,
          "missed_bytes" : 0,
          "history" : "DcA",
          "tunnel_parents" : [ ],
          "message" : "{\"ts\":1487969779.653504,\"uid\":\"CPndOf4NNf9CzTILFi\",\"id_orig_h\":\"192.168.137.130\",\"id_orig_p\":56058,\"id_resp_h\":\"192.168.137.141\",\"id_resp_p\":22,\"proto\":\"tcp\",\"duration\":848.388112,\"orig_bytes\":32052,\"resp_bytes\":0,\"conn_state\":\"OTH\",\"local_orig\":true,\"local_resp\":true,\"missed_bytes\":0,\"history\":\"DcA\",\"orig_pkts\":1364,\"orig_ip_bytes\":102980,\"resp_pkts\":0,\"resp_ip_bytes\":0,\"tunnel_parents\":[]}",
          "tags" : [
            "beats_input_codec_plain_applied"
          ],
          "@timestamp" : "2017-02-24T21:15:29.414Z",
          "resp_bytes" : 0,
          "proto" : "tcp",
          "fields" : {
            "sensorType" : "networksensor"
          },
          "ts" : 1.487969779653504E9
        },
        "sort" : [
          1487970929414
        ]
      }
    ]
  }
}

如您所見，在外部應用程序（用C＃編寫，因此所有這些字符串上的垃圾回收量很大）中要處理的輸出很多 ，而我只是不需要。

我的問題是，如何設置查詢，以便僅獲取所需的字段？

Answer 1

對於5.x，有一項更改使您可以進行_source過濾。 該文檔在這里 ，看起來像這樣：

{ 
 "query": {
   "match_all" : { }
 },
 "size": 1,
 "_source": ["a","b"],
 ...

結果看起來像：

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "xxx",
        "_type" : "xxx",
        "_id" : "xxx",
        "_score" : 1.0,
        "_source" : {
          "a" : 1,
          "b" : "2"
        }
      }
    ]
  }
}

對於5之前的版本，可以使用fields參數來實現：

您的查詢可以在查詢的根級別傳遞,"fields": ["field1","field2"...] 。 它返回的格式會有所不同，但是會起作用。

{ 
"query": {
  "match_all" : { }
},
"size": 1,
"fields": ["a","b"],
...

這將產生如下輸出：

  {
  "took": 9,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "hits": {
    "total": 2077,
    "max_score": 1,
    "hits": [
      {
        "_index": "xxx",
        "_type": "xxx",
        "_id": "xxxx",
        "_score": 1,
        "fields": {
          "a": [
            0
          ],
          "b": [
            "xyz"
          ]
        }
      }
    ]
  }
}

這些字段始終是數組（自1.0 API起），並且沒有任何方法可以更改，因為Elasticsearch本質上是多值感知的。