Logstash grok 過濾器 - 字段值重復

Question

我的logstash過濾器配置如下：

filter {
  grok {
    patterns_dir => ["/usr/share/logstash/pipeline/patterns/"]
    match => {
      "[message]" => "%{TIMESTAMP_ISO8601:timestamp} %{THREAD:thread} %{LOGLEVEL:level} %{LOGGER:logger} %{CONTEXT:context} - %{GREEDYDATA:message}"
    }
  }
  mutate {
    rename => { "[fields][index]" => "application" }
    rename => { "[host][name]" => "instance" }
    remove_field => ["@version","agent.ephemeral_id","agent","ecs","fields","input","tags"]
  }
}

Grok 調試器建議一切正常，對於錯誤行：

2020-10-28 05:14:41,282 [Worker-5] DEBUG Amount  - calculate operation: [1], useCurrencyCodeOfPosition: [false]

我得到以下輸出：

{
  "level": "DEBUG",
  "logger": "Amount",
  "context": "",
  "thread": "Worker-5",
  "message": "calculate operation: [1], useCurrencyCodeOfPosition: [false]",
  "timestamp": "2020-10-28 05:14:41,282"
}

模式定義如下：

THREAD \[(?<thread>[^\]]*)\]
LOGGER (?<logger>[^ ]*)
CONTEXT (?<context>[^-]*)

現在，grok 過濾器產生的每個值都被復制，如下例所示：

             "logger" => [
    [0] "Amount",
    [1] "Amount"
],
             "thread" => [
    [0] "[Worker-5]",
    [1] "Worker-5"

這里有什么問題？ 我只是想不通。 這是我的第一個過濾器:)。 我正在使用 Logstash 7.9.2 (dockerized)

Answer 1

我認為過濾器中的自定義模式存在問題。 只需使用如下開箱即用的模式也可以實現您想要的

filter{
 grok {
    match => {
      "message" => "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}\[%{DATA:thread}\]%{SPACE}%{LOGLEVEL:level}%{SPACE}%{NOTSPACE:logger} %{DATA:context}-%{SPACE}%{GREEDYDATA:message}"
    }
    overwrite => [ "message" ]
  }
  mutate {
    rename => { "[fields][index]" => "application" }
    rename => { "[host][name]" => "instance" }
    remove_field => ["@version","agent.ephemeral_id","agent","ecs","fields","input","tags"]
  }
}

查看此默認 grok 模式的鏈接。 如果您需要對這些事件進行時間序列分析，我建議您使用timestamp覆蓋@timestamp或至少在timestamp上應用日期過濾器。

如果您希望捕獲多行堆棧跟蹤錯誤，請考慮在輸入插件上使用多行過濾器。