Grok parse failure with Logstash

Question

I am trying to parse my nginx logs and send them to ElasticSearch for analysis. This here is my logstash configuration:

logstash.conf

input {
 file {
   path => "/var/log/nginx/access.log"
   type => "nginx_access"
 }
}
filter {
  if [type] == "nginx_access" {
    grok {
      patterns_dir => "/home/daspiyush0/logstash-6.1.2/patterns"
      match => { "message" => "%{NGINX_ACCESS}" }
      remove_tag => ["nginx_access", "_grokparsefailure"]
      add_field => {
        "type" => "nginx_access"
      }
      remove_field => ["program"]
    }

    date {
      match => ["time_local", "dd/MMM/YYYY:HH:mm:ss Z"]
      target => "@timestamp"
      remove_field => "time_local"
    }

    useragent {
      source => "user_agent"
      target => "useragent"
      remove_field => "user_agent"
    }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    manage_template => true
    template_overwrite => true
    template => "/home/daspiyush0/logstash-6.1.2/templates/es_template.json"
    index => "logstash-%{+YYYY.MM.dd}"
  }
}

/home/daspiyush0/logstash-6.1.2/patterns/nginx_access

METHOD (OPTIONS|GET|HEAD|POST|PUT|DELETE|TRACE|CONNECT)
NGINX_ACCESS %{IPORHOST:visitor_ip} - - \[%{HTTPDATE:time_local}\] "%
{METHOD:method} %{URIPATHPARAM:path} HTTP/%{NUMBER:http_version}" %
{INT:status} %{INT:body_bytes_sent} "%{URI:referer}" "%
{QS:user_agent}"

sample nginx log

127.0.0.1 - - [19/Jan/2018:12:03:52 +0530] "GET /favicon.ico HTTP/1.1" 
502 575 "http://127.0.0.1/" "Mozilla/5.0 (X11; Linux x86_64) 
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 
Safari/537.36" "-"

sample document formed

{
  "_index": "logstash-2018.01.19",
  "_type": "nginx_access",
  "_id": "AWENLcPHlWpuWFLYWlZ6",
  "_score": 1,
  "_source": {
    "@version": "1",
    "tags": [
      "_grokparsefailure"
    ],
    "host": "daspiyush0-thinkpad-e450",
    "type": "nginx_access",
    "path": "/var/log/nginx/access.log",
    "@timestamp": "2018-01-19T06:49:17.684Z",
    "message": "127.0.0.1 - - [19/Jan/2018:12:19:17 +0530] \"GET / HTTP/1.1\" 502 575 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36\" \"-\""
  }
}

I am getting a grok parse failure with the above filter configuration. What am I doing wrong?

Answer 1

Your pattern is almost right. Problem is the latest token of it, "% {QS:user_agent}" . QS stands for quoted string , but you surrounded your QS field with another pair of double quotes. Change your pattern as follows and it should work:

 NGINX_ACCESS %{IPORHOST:visitor_ip} - - \[%{HTTPDATE:time_local}\] "%{METHOD:method} %{URIPATHPARAM:path} HTTP/%{NUMBER:http_version}" %{INT:status} %{INT:body_bytes_sent} "%{URI:referer}" %{QS:user_agent}