Logstash过滤器不支持正则表达式

Question

I am reading files as input and thenafter pass it to be filtered, and accordingly based on the [type] the if/else for output (stdout) follows.

here is the conf part :

filter {
    if [path] =~ "error" {
        mutate {
            replace => { "type" => "ERROR_LOGS"}
        }
        grok {
            match => {"error_examiner" => "%{GREEDYDATA:err}"}
        }
        if [err] =~ "9999" {
            if [err] =~ "invalid offset" {
                mutate {
                    replace => {"type" => "DISCARDED_ERROR_LOGS"}
                }
                grok {
                    match => {"message" => "\[%{DATA:date}\] \[%{WORD:logtype} \] \[%{IPORHOST:ip}\]->\[http://search:9999/%{WORD:searchORsuggest}/%{DATA:askme_tag}\?%{DATA:paramstr}\] \[%{DATA:reason}\]"}
                }
                date {
                    match => [ "date", "YYYY-MM-DD aaa HH:mm:ss" ]
                    locale => en
                }
                geoip {
                    source => "ip"
                    target => "geo_ip"
                }
                kv {
                    source => "paramstr"
                    trimkey => "&\?\[\],"
                    value_split => "="
                    target => "params"
                }
            }
            else {
                mutate {
                    replace => {"type" => "ACCEPTED_ERROR_LOGS"}
                }
                grok {
                    match => {
                        "message" => "\[%{DATA:date}\] \[%{WORD:logtype} \] \[%{WORD:uptime}\/%{NUMBER:downtime}\] \[%{IPORHOST:ip}\]->\[http://search:9999/%{WORD:searchORsuggest}\/%{DATA:askme_tag}\?%{DATA:paramstr}\]"
                    }
                }
                date {
                    match => [ "date" , "YYYY-MM-DD aaa HH:mm:ss" ]
                    locale => en
                }
                geoip {
                    source => "ip"
                    target => "geo_ip"
                }
                kv {
                    source => "paramstr"
                    trimkey => "&\?\[\],"
                    value_split => "="
                    target => "params"
                }
            }
        }
        else if [err] =~ "Exception" {
            mutate {
                    replace => {"type" => "EXCEPTIONS_IN_ERROR_LOGS"}
            }
            grok {
                match => { "message" => "%{GREEDYDATA}"}
            }
        }
    }
    else if [path] =~ "info" {
        mutate {
            replace => {"type" => "INFO_LOGS"}
        }
        grok {
            match => {
                "info_examiner" => "%{GREEDYDATA:info}"
            }
        }
        if [info] =~ "9999" {
            mutate {
                replace => {"type" => "ACCEPTED_INFO_LOGS"}
            }
            grok {
                    match => {
                        "message" => "\[%{DATA:date}\] \[%{WORD:logtype} \] \[%{WORD:uptime}\/%{NUMBER:downtime}\]( \[%{WORD:qtype}\])?( \[%{NUMBER:outsearch}/%{NUMBER:insearch}\])? \[%{IPORHOST:ip}\]->\[http://search:9999/%{WORD:searchORsuggest}/%{DATA:askme_tag}\?%{DATA:paramstr}\]"
                    }
            }
            date {
                match => [ "date" , "YYYY-MM-DD aaa HH:mm:ss" ]
                locale => en
            }
            geoip {
                source => "ip"
                target => "geo_ip"
            }
            kv {
                source => "paramstr"
                trimkey => "&\?\[\],"
                value_split => "="
                target => "params"
            }
        }
        else {
            mutate {replace => {"type" => "DISCARDED_INFO_LOGS"}}
            grok {
                match => {"message" => "%{GREEDYDATA}"}
            }
        }
    }
}

the grok regexps I have tested to be working http://grokdebug.herokuapp.com/

however, what's not working is this part :

grok {
            match => {"error_examiner" => "%{GREEDYDATA:err}"}
        }
        if [err] =~ "9999" {

I was wondering what's wrong in there ???

Answer 1

Actually, I have fixed it. Here is what I'd like to share with other fellows, of what I learnt while initial experiments with logstash, for the documentation and other resources aren't so very telling ...

1. "error_examiner" or "info_examiner" wont work, parse the instance/event row in "message"
2. geoip doesnt work for internal ips.
3. kv, for this you must specify the field_split and value_split if they aren't like a=1 b=2 , but say a:1&b:2 then field_Split is &, value_split is :
4. stdout, by default badly prepends if codec chosen is json, please choose rubydebug.

Thanks,