I am reading files as input and thenafter pass it to be filtered, and accordingly based on the [type] the if/else for output (stdout) follows.
here is the conf part :
filter {
if [path] =~ "error" {
mutate {
replace => { "type" => "ERROR_LOGS"}
}
grok {
match => {"error_examiner" => "%{GREEDYDATA:err}"}
}
if [err] =~ "9999" {
if [err] =~ "invalid offset" {
mutate {
replace => {"type" => "DISCARDED_ERROR_LOGS"}
}
grok {
match => {"message" => "\[%{DATA:date}\] \[%{WORD:logtype} \] \[%{IPORHOST:ip}\]->\[http://search:9999/%{WORD:searchORsuggest}/%{DATA:askme_tag}\?%{DATA:paramstr}\] \[%{DATA:reason}\]"}
}
date {
match => [ "date", "YYYY-MM-DD aaa HH:mm:ss" ]
locale => en
}
geoip {
source => "ip"
target => "geo_ip"
}
kv {
source => "paramstr"
trimkey => "&\?\[\],"
value_split => "="
target => "params"
}
}
else {
mutate {
replace => {"type" => "ACCEPTED_ERROR_LOGS"}
}
grok {
match => {
"message" => "\[%{DATA:date}\] \[%{WORD:logtype} \] \[%{WORD:uptime}\/%{NUMBER:downtime}\] \[%{IPORHOST:ip}\]->\[http://search:9999/%{WORD:searchORsuggest}\/%{DATA:askme_tag}\?%{DATA:paramstr}\]"
}
}
date {
match => [ "date" , "YYYY-MM-DD aaa HH:mm:ss" ]
locale => en
}
geoip {
source => "ip"
target => "geo_ip"
}
kv {
source => "paramstr"
trimkey => "&\?\[\],"
value_split => "="
target => "params"
}
}
}
else if [err] =~ "Exception" {
mutate {
replace => {"type" => "EXCEPTIONS_IN_ERROR_LOGS"}
}
grok {
match => { "message" => "%{GREEDYDATA}"}
}
}
}
else if [path] =~ "info" {
mutate {
replace => {"type" => "INFO_LOGS"}
}
grok {
match => {
"info_examiner" => "%{GREEDYDATA:info}"
}
}
if [info] =~ "9999" {
mutate {
replace => {"type" => "ACCEPTED_INFO_LOGS"}
}
grok {
match => {
"message" => "\[%{DATA:date}\] \[%{WORD:logtype} \] \[%{WORD:uptime}\/%{NUMBER:downtime}\]( \[%{WORD:qtype}\])?( \[%{NUMBER:outsearch}/%{NUMBER:insearch}\])? \[%{IPORHOST:ip}\]->\[http://search:9999/%{WORD:searchORsuggest}/%{DATA:askme_tag}\?%{DATA:paramstr}\]"
}
}
date {
match => [ "date" , "YYYY-MM-DD aaa HH:mm:ss" ]
locale => en
}
geoip {
source => "ip"
target => "geo_ip"
}
kv {
source => "paramstr"
trimkey => "&\?\[\],"
value_split => "="
target => "params"
}
}
else {
mutate {replace => {"type" => "DISCARDED_INFO_LOGS"}}
grok {
match => {"message" => "%{GREEDYDATA}"}
}
}
}
}
the grok regexps I have tested to be working http://grokdebug.herokuapp.com/
however, what's not working is this part :
grok {
match => {"error_examiner" => "%{GREEDYDATA:err}"}
}
if [err] =~ "9999" {
I was wondering what's wrong in there ???
Actually, I have fixed it. Here is what I'd like to share with other fellows, of what I learnt while initial experiments with logstash, for the documentation and other resources aren't so very telling ...
Thanks,
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.