Logstash JSON Grok 過濾器問題

Question

我設置了 squid 代理以通過 Logstash 將 JSON 格式的日志發送到 Elastic。 我正在嘗試使用 GROK 過濾來解析日志。 該過濾器在 Kiabana Grok 調試器中工作，但當我重新啟動 Logstash 時抱怨以下錯誤

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:squid_logs,
 :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"
{\", \",\", \"]\" at line 10, column 62 (byte 137) after filter {\n  grok {\n    match => {\n 
       \"message\" => [ \"%{IPV4:vendor_ip}\", \"%{WORD:message}\"", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", 
"org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", 
"org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", 
"/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", 
"/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", 
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:389:in `block in converge_state'"]}

我有以下 GROK 過濾器

"%{IPV4:vendor_ip}", "%{WORD:message}": "%{IPV4:clientip}", "%{WORD:message}": "%
{DATA:timestamp}", "%{WORD:message}": "%{WORD:verb}", "%{WORD:message}": "%{DATA:request}", "%
{WORD:message}": "%{URIPATHPARAM:path}"

在 Kibana Grok 調試器中，過濾器可以很好地處理如下消息：

{ "vendor_ip": "x.x.x.x", "clientip": "x.x.x.x", "timestamp": "2021-04-09T13:58:38+0000", 
"verb": "GET", "request": "https://domain", "path": "/somepath", "httpversion": "HTTP/1.1", 
"response": 200, "bytes": 2518042, "referer": "-", "useragent": "Microsoft BITS/7.8", 
"request_status": "HIER_DIRECT", "hierarchy_status": "HIER_DIRECT" }

Logstash 配置如下：

input {
  beats {
    port => 5045
  }
}

filter {
  grok {
    match => {
        "message" => [ "%{IPV4:vendor_ip}", "%{WORD:message}": "%{IPV4:clientip}", "%{WORD:message}": "%{DATA:timestamp}", "%{WORD:message}": "%{WORD:verb}", "%{WORD:message}": "%{DATA:request}", "%{WORD:message}": "%{URIPATHPARAM:path}" ]
    }
  }
}


output {
  elasticsearch {
    hosts => ["x.x.x.x:9200"]
    index => "squid_logs"
  }
}

Answer 1

使用 grok 過濾器解析 json 消息是錯誤的方法，沒有必要這樣做，而且工作量很大，因為您需要轉義消息中的所有雙引號，否則會出現配置錯誤，即你的情況。

使用json過濾器解析 json 消息

只需在您的管道中使用它：

filter {
    json {
        source => "message"
    }
}