Logstash-Grok-多行异常

Question

I'm trying to parse through a grok filter some very various exception, so I wrote a grok filter, with the help of rubular.com, to parse every single type of exception. The filter is:

grok {
match => { message => "^(?<year>\d{4})-(?<month>\d{1,2})-(?<day>\d{1,2})\W(?<hours>\d{2}):(?<minutes>\d{2}):(?<seconds>\d{2})(,)[0-9]*(.*)(?<log_level>(ERROR|INFO)) (?<exception>(.*\n^Axis.*\n.*\n.*\n.*\n.*\n.*\n.*\n.*)|(com.*trying.*\ncom.*is:.*\n.*java.*)|(com.*\n^org.*\n###.*non valido\n\n.*^###.*\n^###.*\n^###.*)|(.*trying.*\n^com.*ServiceException.*\n### Error querying.*\n\n.*\n^###.*\n.*)|(.*trying.*\n^com.*ServiceException.*\n^###.*\n^###.*)|(.*trying.*\n^com.*)|(.*\n^org.*\n###.*Exception.*\n### Cause:.*)|(com.*\n^org.*\n###.*)|(.*\n^java.*CORBA.*\n.*)|(.*\n^java*.*)|(com.*\n^com.*)|(.*null\n^Axis.*\n.*\n.*\n.*\n.*\n.*\n.*\n.*\n.*\n.*\n.*)|(.*\n))"}
}

which as you can see as a lot of OR conditions in the exception field and a lot of \\n to take the carriage returns. The problem is that, from what I understood, Logstash can read only one line at a time and can't match multiple lines (so, even if on rubular this pattern was working perfectly, it doesn't in logstash). How can I filter the exceptions correctly?

Answer 1

You can multiline before grok, for example java exceptions:

multiline {
    type => %sometype
    pattern => "(^\s)"
    what => previous 
          }

So this will append all lines that starts with whitespace to previous. And after that you can use grok filter.

Oh, and you can mutate to avoid '\\n' symbols after multiline:

mutate {
    gsub => ["message", "\n", " "]
       }

After that you are ready to filter multiline message.