Logstash - grok multiline

Question

I tried using multiline in grok filters but its not working properly.

My Logs are

H3|15:55:04:760|exception|not working properly
message:space exception
 at line number 25

My conf file is

input { file {

    path => "logs/test.log"
    start_position => beginning
    sincedb_path => "/dev/null"
  }}
filter{

 multiline {

    pattern => "^(\s|[A-Z][a-z]).*"
    what => "previous"
  }
if [message] =~ /H\d+/{

grok {

match => ["message", "(?m)%{USERNAME:level}\|%{TIME:timestamp}\|%{WORD:method}\|%{GREEDYDATA:error_Message}" ]
  }
   }

   else {

   grok {

match => ["message", "(?m)%{GREEDYDATA:error_Message}" ]
  }
   }
  }

output {elasticsearch { host => "localhost"  protocol => "http" port => "9200" }}

I am able to process the first line of log file, but second line of log file is not working where I would like to use multiline

Output i would like to have

{

"@timestamp" => "2014-06-19 00:00:00,000"
"path" => "logs/test.log"
"level"=>"H3"
"timestamp"=>15:55:04:760
"method"=>exception
"error_message"=>not working properly
},
{
"@timestamp" => "2014-06-19 00:00:00,000"
"path" => "logs/test.log"
"error_message" => "space exception at line 25"
}   

Kindly help me to get required output.

Answer 1

Your multiline config says, "if I find this pattern, keep it with the previous line".

Your pattern "^(\\s|[AZ][az]).*" says "either a space, or a capital letter followed by a lowercase letter, then followed by other stuff".

So, " foo" or "California" would match, but "H3" wouldn't.

I would suggest a pattern that matches the start of your multiline expression, and use the 'negate' feature to have all lines that don't match that pattern join to the original line:

filter {
    multiline {
      pattern => "^[A-Z][0-9]\|"
      negate => 'true'
      what => 'previous'
    }
  }
}

This would take the "H3|" line as the beginning, and join all other lines to it. Depending on the range of values at the beginning of the line, you may need to edit the regexp.