Logstash grok pattern to filter custom Log message

Question

I am new to logstash and I want to filter fileds from log message. Here is log message:

[2015-03-16 13:12:05,130]  INFO - LogMediator ServiceName = TestService_v1,SystemDate = 3/16/15 1:12 PM,ServerIP = 127.0.1.1,ServerHost = Inspiron-3521,SequenceName = Validation,Message = Going to Validate Request ,MessageCode = null,ErrorMessage = null,ErrorDetail = null,ErrorException = null

From above log message I want to extract all fields, like ServiceName, SystemDate, SequenceName etc. What will be the grok pattern or Regex for this log message?

Any help would be appreciated.

Answer 1

you could first split your message in three parts (timestamp, loglevel and the remainding logdata) using:

\[%{TIMESTAMP_ISO8601:timestamp}\]\s+%{WORD:loglevel}\s+-\s+%{GREEDYDATA:logData}

You could then apply the csv filter to the logdata field like so:

csv {
  columns => ["serviceName","systemDate","serverIP","serverHost","sequenceName","message","messageCode","errorMessage","errorDetail","errorException"]
  separator => ","
}

This will split your logData after each , So you would get a new field named message containing the text "Message = Going to Validate Request" You could now edit the individual fields for instance you could extract the actual message using the following grok filter:

Message = %{GREEDYDATA:messageText}

I've found it very helpful to use the grok debugger to work out the individual grok patterns: http://grokdebug.herokuapp.com/