Logstash - grok use a field other than message

Question

I am receiving Log4j generated log files from remote servers using Logstash forwarder. The log event has fields including a field named "file" in the format /tomcat/logs/app.log, /tomcat/logs/app.log.1, etc. Of course file path /tomcat/logs is on the remote machine and I would like Logstash to create files on the local file system using only the file name and not use the remote file path.

Locally, I would like to create a file based on file name app.log, app.log.1, etc. How can one accomplish this?

I am unable to use grok since it appears to work only with "message" field and not others.

Example Log Event:

{"message":["11 Sep 2014 16:29:04,934 INFO LOG MESSAGE DETAILS HERE "],"@version":"1","@timestamp":"2014-09-15T05:44:43.472Z","file":["/tomcat/logs/app.log.1"],"host":"aus-002157","offset":"3116","type":"app.log"}

Logstash configuration - what do I use to write the filter section?

input {
    lumberjack {
      port => 48080
      ssl_certificate => "/tools/LogStash/logstash-1.4.2/ssl/logstash.crt"
      ssl_key => "/tools/LogStash/logstash-1.4.2/ssl/logstash.key"
    }
}

filter{

}

output {

    file{
        #message_format => "%{message}"
        flush_interval => 0
        path => "/tmp/%{host}.%{type}.%{filename}"
        max_size => "4M"
    }
}

Answer 1

Figured out the pattern to be as follows:

grok{
    match => [ "file",  "^(/.*/)(?<filename>(.*))$" ]
}

Thanks for the help!

Answer 2

Logstash Grok can parse all the fields in a log event, not only message field.

For example, you want to extract the file field, you can do like this

filter {
   grok {
       match => [ "file", "your pattern" ]
   }
}