Please help, I'm trying to add grok filter in my Logstash pipeline which will convert below logline
2020-11-06 12:57:43,854 INFO Bandwidth: NASDAQ:224.0.130.65:30408 0.000059 Gb/S
to
{
"ts": [
[
"2020-11-06 12:57:43,854"
]
],
"YEAR": [
[
"2020"
]
],
"MONTHNUM": [
[
"11"
]
],
"MONTHDAY": [
[
"06"
]
],
"HOUR": [
[
"12",
null
]
],
"MINUTE": [
[
"57",
null
]
],
"SECOND": [
[
"43,854"
]
],
"ISO8601_TIMEZONE": [
[
null
]
],
"loglevel": [
[
"INFO"
]
],
"Metric": [
[
"Bandwidth"
]
],
"Chanel": [
[
"NASDAQ:224.0.130.65:30408"
]
],
"Data": [
[
"0.000059 Gb/S"
]
]
}
and below is my grok filter
input{
beats{
port => "5044"
}
}
filter{
if "Bandwidth" in [message]{
grok{
match => {"message" => "%{TIMESTAMP_ISO8601:ts} %{LOGLEVEL:loglevel} %{WORD:Metric}: (?<Chanel>[A-Z]+:[0-9]+.[0-9]+.[0-9]+.[0-9]+:[0-9]+)"}
}
}
}
output{
elasticsearch{
hosts => [ "localhost:9200" ]
}
}
This filter works perfectly fine when I try it in Grok debugger but not in Logstash when viewed in Kibana. I don't see any name captures from filter. Just the message. If I remove the regex part of filter and add GREEDYDATA, everything works. I'm sure I'm doing something wrong in Regex part.
Your regex pattern is correct and does give the expected filter output. Refresh your index pattern in Kibana or try re-ingesting the data.
Although, I do not think you need to use regex if the channel pattern is going to be like Some Data:IP Address:Port
Try below pattern
grok{
match => { "message" => ["%{TIMESTAMP_ISO8601:ts} %{LOGLEVEL:loglevel} %{WORD:Metric}: (?<Channel>%{DATA}:%{HOSTPORT}) (?<Data>%{GREEDYDATA})"]}
}
Logstash output will be
{
"ts" => "2020-11-06 12:57:43,854",
"Metric" => "Bandwidth",
"@timestamp" => 2020-11-06T22:47:20.383Z,
"loglevel" => "INFO",
"host" => "e7c15acec470",
"Data" => "0.000059 Gb/S",
"Channel" => "NASDAQ:224.0.130.65:30408",
"@version" => "1",
"message" => "2020-11-06 12:57:43,854 INFO Bandwidth: NASDAQ:224.0.130.65:30408 0.000059 Gb/S"
}
Try using stdout output along with elasticsearch so you can see what logstash is outputting to elastic.
output{
stdout { codec => rubydebug }
}
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.