logstash / grok pattern file

Question

I am parsing IIS logs, and I have everything working when all patterns are in the config file.

I want to take out all the patterns and put them in a pattern file, but cannot seem to get it to work.

What I have: Log example:

2015-09-08 16:02:23 GET /l8Wc2pt1FMvzsCEJ/test/restapiname

2015-09-08 16:02:24 GET /l8Wc2pt1FMvzsCEJ/test/ifSoap/soapapiname grok which works:

match => { "message" => [
        "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:method} \/%{WORD:orgid}\/(?i)test\/%{GREEDYDATA:restapiname}",
        "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:method} \/%{WORD:orgid}\/(?i)test\/(?i)ifsoap\/%{GREEDYDATA:soapapiname}"

This works. But I have way too many combinations of this url, and want to take the full thing and put it in a file, so I only have to maintain 1 file.

This does not seem to work

patterns file:

IISLOGS %{TIMESTAMP_ISO8601:log_timestamp} %{WORD:method} \/%{WORD:orgid}\/(?i)test\/%{GREEDYDATA:restapiname}
IISLOGS %{TIMESTAMP_ISO8601:log_timestamp} %{WORD:method} \/%{WORD:orgid}\/(?i)test\/(?i)ifsoap\/%{GREEDYDATA:soapapiname}"

GROK file:

grok {    
    patterns_dir => "C:/LogProject/LogStash/patterns"
    match => [ "message", "IISLOGS" ]
  } 

Any suggestions?

Answer 1

I personally would recommend to stay with the patterns inside the logstash configuration. An extra patterns file is annoying and harder to maintain in my opinion. However, if you want to use the patterns file for some reason, here is a possible way:

The problem is that you have two different definitions for IISLOGS inside your grok patterns file. You can split the different path formats into multiple patterns and do a logical or inside your IISLOGS definition with (?:%{IISPATH1}|%{IISPATH2}) .

Patterns file:

IISPATH1 \/%{WORD:orgid}\/(?i)test\/(?i)ifsoap\/%{GREEDYDATA:soapapiname}
IISPATH2 \/%{WORD:orgid}\/(?i)test\/%{GREEDYDATA:restapiname}
IISLOGS %{TIMESTAMP_ISO8601:log_timestamp} %{WORD:method} (?:%{IISPATH1}|%{IISPATH2})

This works in grok debugger for your given examples. First results in restapiname: restapiname and second in soapapiname: soapapiname .