ELK-Stack：使用Logstash仅解析Syslog中的IP / MAC

Question

I'm searching for a way to parse IP addresses and MACs from a syslog entry with Logstash. Currently I try to fetch it with GROK, but the problem is, that I might have to match the entire line, instead of just a part of the message itself.

For example I have to following line:

Apr 9 12:41:01 cn1Label=Host ID dvchost=exch01 TrendMicroDsTenant=Primary TrendMicroDsTenantId=0 dstMAC=55:C0:A8:55:FF:41 srcMAC=CA:36:42:B1:78:3D TrendMicroDsFrameType=IP src=10.0.251.84 dst=56.19.41.128 out=166 cs3= cs3Label=Fragmentation Bits proto=ICMP srcPort=0 dstPort=0 cnt=1 act=IDS:Reset cn3=0 cn3Label=DPI Packet Position cs5=0 cs5Label=DPI Stream Position cs6=0 cs6Label=DPI Flags

I wanna fetch the "src" and "dst" IPs and the "srcMAC" and "dstMAC" as well. I would try it like that in Logstash:

grok{
  match => { "message" => "src=%{IPV4:src_ip}" }
  match => { "message" => "dst=%{IPV4:dst_ip}" }
  match => { "message" => "srcMAC=%{MAC:src_mac}" }
  match => { "message" => "dstMAC=%{MAC:dst_mac}" }
}

But it does not work, because it does not match the whole line. I tried with .* and other matching techniques as well, without success.

Is there a way to just parse the IPs like shown without parsing the full line?

I would try to parse other parts of the message, such as protocol as well. The reason why I do not match the full line is, that the some messages are different and need then also another way to extract its values.

Thank you!

Answer 1

You can use the kv filter to deal with key-value pairs like like those you have in your log. To only keep the relevant pairs, use the include_keys option.

In your case, it would look like this:

kv{
    include_keys => [ "src", "dst", "srcMAC", "dstMAC" ]
}

Which would result in:

{
  "dst": "56.19.41.128",
  "host": "frsred-0077",
  "srcMAC": "CA:36:42:B1:78:3D",
  "dstMAC": "55:C0:A8:55:FF:41"
}

One benefit of the kv filter is that you're not dependent on the order of the pairs staying the same, unlike with the grok filter.

Answer 2

The grok filter needs to match the whole message, to fetch only a couple of fields you still need to match everything, the following pattern will match your example.

%{GREEDYDATA}%{SPACE}dstMAC=%{MAC:dst_mac}%{SPACE}srcMAC=%{MAC:src_mac}%{SPACE}%{GREEDYDATA}%{SPACE}src=%{IP:src_ip}%{SPACE}dst=%{IP:dst_ip}%{SPACE}%{GREEDYDATA}

The result will be:

{
  "src_ip": "10.0.251.84",
  "src_mac": "CA:36:42:B1:78:3D",
  "dst_mac": "55:C0:A8:55:FF:41",
  "dst_ip": "56.19.41.128"
}

This pattern will also match any message with the following format:

ANYTHING dstMAC=MACADDRESS srcMAC=MACADDRESS ANYTHING src=IPADDRESS dst=IPADRESS ANYTHING

Answer 3

I just found the solution. I did something very wrong. You have to do a matching filter for each matching separately. If I do so, then I can extract also the content within the message field, for example like:

grok{match => {"message" => "SRC=%{IPV4:ip}"}}