Filter Apache Error Logs in Logstash using grok

Question

I need to filter apache error logs using grok. Please help me with the pattern I am not able to create..

My sample log:

2020-10-07T01:21:26.403-0400    ERROR   [reload]        cfgfile/list.go:96  Error creating runner from config: Error getting config for fileset system/auth: Error interpreting the template of the inp$
2020-10-07T01:21:36.404-0400    ERROR   [reload]        cfgfile/list.go:96  Error creating runner from config: Error getting config for fileset system/auth: Error interpreting the template of the inp$
2020-10-07T01:21:38.925-0400    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://IP:5044)): dial tcp IP:5044: i/o timeout
2020-10-07T01:21:38.925-0400    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(async(tcp://IP:5044)) with 26743 reconnect attempt(s)
2020-10-07T01:21:38.925-0400    INFO    [publish]   pipeline/retry.go:189   retryer: send unwait-signal to consumer
2020-10-07T01:21:38.925-0400    INFO    [publish]   pipeline/retry.go:191     done
2020-10-07T01:21:38.925-0400    INFO    [publish]   pipeline/retry.go:166   retryer: send wait signal to consumer
2020-10-07T01:21:38.925-0400    INFO    [publish]   pipeline/retry.go:168     done

I understand through grok patterns, we can use these below, but I am not understanding how to use this in Grok Pattern:

# Error logs
HTTPD20_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:message}
HTTPD24_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}(:tid %{NUMBER:tid})?\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_message}:)?( \[client %{IPORHOST:clientip}:%{POSINT:clientport}\])?( %{DATA:errorcode}:)? %{GREEDYDATA:message}
HTTPD_ERRORLOG %{HTTPD20_ERRORLOG}|%{HTTPD24_ERRORLOG}

Can anyone please help! Thanks in advance!

Answer 1

After work on your sample data this grok pattern must work:

filter {
    grok {
            match => { "message" => "%{TIMESTAMP_ISO8601}%{SPACE}%{LOGLEVEL}(%{SPACE}\[%{WORD:action}\])?%{SPACE}%{WORD:package}/%{WORD:class}.go:%{INT:line:number}%{SPACE}%{GREEDYDATA:message}$" }
    }
}

Your data are not exactly http, so a custom pattern needed, I guess my grok must be more easy to read without space, I recommand to you to use mutate-gsub to uniformize space (be carreful with your final sentence named "message" in my solution).

You have more details about this pattern and other here .