Logstash grok 過濾器 grok 解析失敗

Question

我使用 Logstash 和 Filebeat 對其進行了配置。 我使用了 MongoDB output 插件。

我想像這樣在 mongoDB 中制作數據：

  {
    "_id": {"$oid": "62ea3197736a54952d56e2fe"},
    "category": "TEST",
    "date": "2022-08-03T17:28:07+09:00",
    "level": "WARNING",
    "message": "statistics_sales.go:136: sdb.FindUserByID, error:  mongo: no documents in result",
    "title": "statistics_sales"
  }

這是原始日志數據：

ERROR: 2022/08/16 13:27:42.292305 statistics_sales.go:136: sdb.FindUserByID, error:  mongo: no documents in result

所以我像這樣配置logstash.conf：

filter {
  grok {
    match => {
      "message" => "%{LOGLEVEL:level}: %{YEAR:year}/%{MONTHNUM:month}/%{MONTHDAY:day} %{TIME:time} %{GREEDYDATA:message}"
    }

    overwrite => [ "message" ]
  }

  grok {
    match => {
      "message" => "%{DATA:title}\.go%{GREEDYDATA}"
    }
  }

  grok {
    match => {
      "time" => "%{DATA:sec}\.%{GREEDYDATA}"
    }
  }

  grok {
    add_field => {
      "date" => "%{year}-%{month}-%{day}T%{time}+09:00"
    }
  }

  grok {
    remove_field => ["year", "month", "day", "time", "event", "@version", "agent", "host", "@timestamp", "ecs"]
  }
}

但是，我在 time(sec) 配置下遇到錯誤。

我嘗試了很多方法，但我找不到問題所在。

這是 Logstash 結果：

{
         "month" => "08",
         "input" => {
        "type" => "log"
    },
         "agent" => {
                "name" => "HOSTNAME",
             "version" => "8.3.3",
                  "id" => "8a97df17-1977-472f-b6f1-6c7a7359c372",
                "type" => "filebeat",
        "ephemeral_id" => "34333de9-c892-42a0-bc35-ee49a0f06721"
    },
           "log" => {
        "offset" => 575,
          "file" => {
            "path" => "/home/bjkang/LogCollector/Log/service_log_2022-08-23.log"
        }
    },
           "ecs" => {
        "version" => "8.0.0"
    },
         "title" => "statistics_sales",
    "@timestamp" => 2022-08-23T11:30:42.407Z,
          "time" => "13:27:42.293679",
           "sec" => "13:27:42",
          "host" => {
        "name" => "HOSTNAME"
    },
          "tags" => [
        [0] "_grokparsefailure"
    ],
      "@version" => "1",
       "message" => "statistics_sales.go:136: sdb.FindUserByID, error:  mongo: no documents in result",
           "day" => "16",
         "level" => "ERROR",
          "year" => "2022",
         "event" => {
        "original" => "ERROR: 2022/08/16 13:27:42.293679 statistics_sales.go:136: sdb.FindUserByID, error:  mongo: no documents in result"
    }
}

我能得到一些提示嗎？

Answer 1

我通過這樣設置解決了它：

filter {
  grok {
    match => {
      "message" => "%{LOGLEVEL:level}: %{YEAR:year}/%{MONTHNUM:month}/%{MONTHDAY:day} %{TIME:time} %{GREEDYDATA:message}"
    }

    overwrite => [ "message" ]
  }

  grok {
    match => {
      "message" => "%{DATA:title}\.go%{GREEDYDATA}"
    }
  }

  grok {
    match => {
      "time" => "%{DATA:hhmmss}\.%{GREEDYDATA}"
    }
  }
  mutate {
    add_field => {
      "date" => "%{year}-%{month}-%{day}T%{hhmmss}+09:00"
    }
  }

  prune {
    whitelist_names => ["level", "message", "title", "date", "@timestamp", "@version"]
  }
}

但是，我無法插入沒有元數據的文檔 [@timestamp, @version]

也許有人知道這一點，請評論:)