在Logstash管道中使用Elasticsearch过滤器

Question

我在logstash管道中使用了Elasticsearch过滤器。 我使用以下命令正确找到结果：

filter{
  if [class] == "DPAPIINTERNAL" {
    elasticsearch {
      hosts => "10.1.10.16"
      index => "dp_audit-2017.02.16"
      query_template => "/home/vittorio/Documents/elastic-queries/matching-requestaw.json"
    }
  }
}

如您所见，我使用“ query_template”，即：

{
    "query": {
      "query_string": {
       "query": "class:DPAPI AND request.aw:%{[aw]}"
      }
    },
   "_source": ["end_point", "vittorio"]
 }

告诉elastichsearch使用与DPAPIINTERNAL日志匹配“ aw”的特定类查找日志。

完善！ 但现在我找到了结果，因此我想从中添加一些字段并将其附加到我的DPAPIINTERNAL日志中，例如，我想获取“ end_point”并将其添加到日志中的新键“ vittorio”中。

这没有发生，我不明白为什么。

这是我正在使用查询查看的日志：

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 1,
    "hits": [
      {
        "_index": "dp_audit-2017.02.16",
        "_type": "logs",
        "_id": "AVpHoPHPuEPlW12Qu",
        "_score": 1,
        "_source": {
          "svc": "dp-1.1",
          "request": {
            "method": "POST|PATCH|DELETE",
            "aw": "prova",
            "end_point": "/bank/6311",
            "app_instance": "7D1-D233-87E1-913"
          },
          "path": "/home/vittorio/Documents/dpapi1.json",
          "@timestamp": "2017-02-16T15:53:33.214Z",
          "@version": "1",
          "host": "Vito",
          "event": "bank.add",
          "class": "DPAPI",
          "ts": "2017-01-16T19:20:30.125+01:00"
        }
      }
    ]
  }
}

Answer 1

您需要在elasticsearch过滤器中指定fields参数 ，如下所示：

elasticsearch {
  hosts => "10.1.10.16"
  index => "dp_audit-2017.02.16"
  query_template => "/home/vittorio/Documents/elastic-queries/matching-requestaw.json"
  fields => { "[request][end_point]" => "vittorio" }
}

请注意，由于end_point是一个嵌套字段，因此您需要像这样修改查询模板中的_source ：

"_source": ["request.end_point"]

Answer 2

问题很简单，您不必使用query_template指定“新”字段。

"_source": ["request"] # here you specify the field you want from the query result.

接着

filter{
  if [class] == "DPAPIINTERNAL" {
    elasticsearch {
      hosts => "10.1.10.16"
      index => "dp_audit-2017.02.16"
      query_template => "/home/vittorio/Documents/elastic-queries/matching-requestaw.json"
      fields => {"request" => "new_key"} # here you add the fields and will tell elastich filter to put request inside new_key
    }
  }
}

那对我有用！