如何使用grok转换logstash中的日志

Question

Here is my log format:

TIME|HOST-IP|REQUEST-ID|UID|USERNAME|USER-AGENT|METHOD|URI|CONTROLLER-METHOD|PARAMS-MAP|RESPONSE-CODE

And, log looks like:

2016-11-12 21:02:45.878|192.168.31.205|368284310235387-20|1|wangziyi|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36|GET|/api/v1|com.didi.km.api.controller.api.v1.IndexController#index[1 args]|{"hi":["asd"]}|200
2016-11-12 21:02:46.100|192.168.31.205|368284310235387-21|1|wangziyi|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36|GET|/api/v1|com.didi.km.api.controller.api.v1.IndexController#index[1 args]|{"hi":["asd"]}|200
2016-11-12 21:02:46.389|192.168.31.205|368284310235387-22|1|wangziyi|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36|GET|/api/v1|com.didi.km.api.controller.api.v1.IndexController#index[1 args]|{"hi":["asd"]}|200
2016-11-12 21:02:46.507|192.168.31.205|368284310235387-23|1|wangziyi|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36|GET|/api/v1|com.didi.km.api.controller.api.v1.IndexController#index[1 args]|{"hi":["asd"]}|200

Logstash version is 5.0.0, Log was send by Filebeat on server.

My logstash config is:

input {
    beats {
        port => "5043"
    }
}
filter {
    grok {
        match => { "message" => "%{WORD:time}|%{IP:hostIP}|%{WORD:requestId}|%{NUMBER:uid:int}|%{WORD:username}|%{WORD:method}|%{URIPATHPARAM:uri}|%{WORD:cont
rollerMethod}|%{WORD:params}" }
    }
}
output {
    stdout { codec => rubydebug }
    elasticsearch {
        hosts => [ "10.94.66.193:9200" ]
    }
}

But, I just get the message field only:

{
    "@timestamp" => 2016-11-12T13:02:48.607Z,
        "offset" => 31831,
      "@version" => "1",
    "input_type" => "log",
          "beat" => {
        "hostname" => "localhost",
            "name" => "localhost",
         "version" => "5.0.0"
    },
          "host" => "localhost",
        "source" => "logs/km-access.2016-11-12.log",
          "time" => "2016",
       "message" => "2016-11-12 21:02:46.507|192.168.31.205|368284310235387-23|1|wangziyi|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36|GET|/api/v1|com.didi.km.api.controller.api.v1.IndexController#index[1 args]|{\"hi\":[\"asd\"]}|200",
          "type" => "log",
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ]
}

Is there some issue in my config code ?

Answer 1

use the follow pattern to match this log:

%{TIMESTAMP_ISO8601}\|%{IP}\|(?<requestID>\d+-\d+)\|%{INT:uid}\|%{WORD:username}\|(?<ua>(\w|\/|\.|\s|\(|;|\)|,)+)\|%{WORD:method}\|(?<uri>(\w|\/)+)\|(?<controllerMethod>(\w|\d|\s|\.|#|\[|\])+)\|(?<param>(\w|{|}|"|\:|\[|\])+)\|%{NUMBER:statusCode}\

And, thanks https://grokdebug.herokuapp.com/