Configuring Logstash filters

Question

I have recently configured a ELK server,

In my app server (magento) var/log/ directory has many log files (including some 3rd party extension logs for magento), So I thought of sending all using *.log to logstash, because we are not aware of some of the log file names that might create in future because of 3rd party integration. and we need to catch those also.

Magento exception log has a multi-line logs with the stack trace, So What i did was add a filter using gork, Now It seems working (giving the concatenate output),

Since I have one define type "staging-all-lincraft-logs" (in both config) all the log files are parsing through it (see the below code),

I can't remove the *.log and give specific names sine un-aware of file names

Is there any way I can parse only the specific files(exception.log and system.log) in logstash config (I tried with adding the parth,It does not work)

logstash forwarder config:

"files": [
    {
      "paths": [
        "/home/deploy/lindcraft/current/codepool/var/log/*.log"
       ],
      "fields": { "type": "staging-all-lincraft-logs" }
    }
]

logstash filter config:

filter {
  if [type] == "staging-all-lincraft-logs" {

    multiline{
#      path => "/home/deploy/lindcraft/current/codepool/var/log/exception.log"
      pattern => "^%{TIMESTAMP_ISO8601:timestamp}"
      what => "previous"
      negate=> true
    }
    grok {
      match => [
        "message",
        "(?m)%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:priority_name} \(%{INT:priority_level}\): %{GREEDYDATA:message}"
      ]
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
      overwrite => [ "message" ]
    }
    date {
      match => [ "timestamp", "ISO8601" ]
    }
  }
}

Answer 1

There is a workaround for you if you can change the log directory of 3rd party extensions. Then you can send all name unaware files to that new directory (eg. /var/log/new_directory/) and keep all other known logs in /var/log/ . So you change your logstash-forwader.conf file as below to send different types of log files to the logstash server.

    "files": [
        {
          "paths": [
              "/home/deploy/lindcraft/current/codepool/var/log/syslog"
           ],
          "fields": { "type": "syslog" }
        },
        {
          "paths": [
              "/home/deploy/lindcraft/current/codepool/var/log/exeption.log"
           ],
           "fields": { "type": "exeption-log" }
        },
        {
          "paths": [
              "/home/deploy/lindcraft/current/codepool/var/log/new_directory/*.log"
           ],
           "fields": { "type": "staging-all-lincraft-logs" }
        }
    ]

And then add a filter to parse different log files with relevant GROK patterns. It will be some thing like below.

filter {
  if [type] == "syslog"  {
      grok {
          match => { "message" => "GROK pattern for syslog" } 
      }
  }
  else if [type] == "exeption-log" {
      grok {
          match => { "message" => "GROK pattern for exeption-log" } 
      }
  }
  else if [type] == "staging-all-lincraft-logs" {
      grok {
          match => { "message" => "GROK pattern for staging-all-lincraft-logs" } 
      }
  }
}

I think if we need to treat log files differently in the logstash, we have to send them with specific types from logstash-forwarder.

Hope it helps.!