Logstash JSON Grok filter issue

Question

I have squid proxies setup to send JSON formatted logs to Elastic via Logstash. I am trying to use GROK filtering to parse the logs. The filter works in the Kiabana Grok Debugger but complains with the following error when I restart Logstash

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:squid_logs,
 :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"
{\", \",\", \"]\" at line 10, column 62 (byte 137) after filter {\n  grok {\n    match => {\n 
       \"message\" => [ \"%{IPV4:vendor_ip}\", \"%{WORD:message}\"", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", 
"org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", 
"org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", 
"/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", 
"/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", 
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:389:in `block in converge_state'"]}

I have the following GROK filter

"%{IPV4:vendor_ip}", "%{WORD:message}": "%{IPV4:clientip}", "%{WORD:message}": "%
{DATA:timestamp}", "%{WORD:message}": "%{WORD:verb}", "%{WORD:message}": "%{DATA:request}", "%
{WORD:message}": "%{URIPATHPARAM:path}"

In the Kibana Grok Debugger the filter works fine against a message like the following:

{ "vendor_ip": "x.x.x.x", "clientip": "x.x.x.x", "timestamp": "2021-04-09T13:58:38+0000", 
"verb": "GET", "request": "https://domain", "path": "/somepath", "httpversion": "HTTP/1.1", 
"response": 200, "bytes": 2518042, "referer": "-", "useragent": "Microsoft BITS/7.8", 
"request_status": "HIER_DIRECT", "hierarchy_status": "HIER_DIRECT" }

Logstash config below:

input {
  beats {
    port => 5045
  }
}

filter {
  grok {
    match => {
        "message" => [ "%{IPV4:vendor_ip}", "%{WORD:message}": "%{IPV4:clientip}", "%{WORD:message}": "%{DATA:timestamp}", "%{WORD:message}": "%{WORD:verb}", "%{WORD:message}": "%{DATA:request}", "%{WORD:message}": "%{URIPATHPARAM:path}" ]
    }
  }
}


output {
  elasticsearch {
    hosts => ["x.x.x.x:9200"]
    index => "squid_logs"
  }
}

Answer 1

Parsing a json message using the grok filter is the wrong approach, there is no need to do this and it will be a lot of work since you will need to escape all the double quotes in the message or you will get configuration errors, which is your case.

Use the json filter to parse json messages

Just use this in your pipeline:

filter {
    json {
        source => "message"
    }
}