How to handle multiple inputs with Logstash in the same file?

Question

Let's say you have very 3 different lines in your log firewall file and you want:

to grok it and the result be stored into an elastic search cluster using the dedicated elastic search output.

what should i do in my logstash.conf ??

Thanks.

Answer 1

Assuming the different logs come from the same log source (ie the same file) and should be regarded as being of the same type (which is judgment call) you can just list multiple grok patterns:

filter {
  grok {
    match => ["message", "pattern1", "pattern2", ..., "patternN"]
  }
}

Listed patterns will be tried in order.

If log messages come from different inputs and are completely different, use the type field to distinguish between the different messages:

filter {
  if [type] == "foolog" {
    grok {
       match => ["message", "pattern1"]
    }
  } else if [type] == "barlog" {
    grok {
       match => ["message", "pattern2"]
    }
  }
}

This pattern might be appropriate also for messages coming from the same input, but that takes a bit more work since you first have to examine the message in a conditional to determine which type to pick.

Answer 2

looking at your comments under Magnus post, I can share maybe a bit more specific example.

Option 1) The logs from our Fortigate are similar and the related grok filter looks like this:

    grok {
        match => [
            "message" , "%{FORTIGATE_50_BASE} %{FORTIGATE_50_V1}",
            "message" , "%{FORTIGATE_50_BASE} %{FORTIGATE_50_V2}",
            "message" , "%{FORTIGATE_50_BASE} %{FORTIGATE_50_V3}",
            "message" , "%{FORTIGATE_50_BASE}"
        ]
        tag_on_failure => [ "failure_grok_fortigate" ]
        break_on_match => false
    }

And the related patterns are these:

   FORTIGATE_50_BASE %{SYSLOG5424PRI:syslog_index}date=%{FORTIDATE:date} time=%{TIME:time} devname=%{HOST:hostname} devid=%{HOST:devid} logid=%{NUMBER:logid} type=%{WORD:fortigate_type} subtype=%{WORD:subtype} level=%{WORD:loglevel} vd=\"?%{WORD:vdom}\"?
   FORTIGATE_50_V1 srcip=%{IP:srcip} srcintf=\"%{HOST:srcintf}\" dstip=%{IP:dstip} dstintf=\"%{HOST:dstintf}\" sessionid=%{NUMBER:sessionid} status=%{DATA:status} policyid=%{DATA:policyid} dstcountry=\"%{DATA:dstcountry}\" srccountry=\"%{DATA:dstcountry}\" trandisp=%{WORD:trandisp} service=%{WORD:service} proto=%{INT:proto} app=%{WORD:app} duration=%{INT:duration} sentbyte=%{INT:sentbyte} rcvdbyte=%{INT:rcvdbyte} sentpkt=%{INT:sentpkt} rcvdpkt=%{INT:rcvdpkt}
   FORTIGATE_50_V2 user=\"%{PROG:user}\" ui=%{GREEDYDATA:ui} msg=\"%{GREEDYDATA:msg}\"
   FORTIGATE_50_V3 action=\"%{PROG:action}\" tunneltype=\"%{PROG:tunneltype}\" tunnel_id=%{NUMBER:tunnel_id} remote_ip=(%{IP:remote_ip}|\(null\)) tunnel_ip=(%{IP:tunnel_ip}|\(null\)) user=\"%{PROG:user}\" group=\"%{PROG:group}\" dst_host=\"%{PROG:dst_host}\" reason=\"%{PROG:reason}\" msg=\"%{GREEDYDATA:msg}\"

So there's a base pattern shared between all logs and some specific parts and the way the grok filter is configured is to handle the more specific patterns first and fall back to the generic one.

Option 2) You logs look very much like key value lists, so maybe a simpler kv filter will help you much more than any pattern.

In your case it might be as simple as:

input {
 ...
}
filter {
  kv { }
  date { ... pull in the date data ... }
}
output {
  ..
}

Hope any of that works. In general I also found it very helpful to have some unittests for ever now log format available.

Cheers