I have some problems while using logstash to opensearch.
filter{
grok {
patterns_dir => ["/etc/logstash/conf.d/patterns"]
match => [ "message","%{DATE_FORM:logdate}%{LOGTYPE:logtype}:%{SPACE}%{GREEDYDATA:msgbody}" ]
}
date {
match => ["logdate", "yyyy.MM.dd-HH.mm.ss:SSS"]
timezone => "UTC"
target=>"timestamp"
}
mutate {
remove_field => ["message"]
add_field => {
"file" => "%{[@metadata][s3][key]}"
}
}
}
This is the conf file I'm using for logstash. In the opensearch console
@timestamp : Dec 15, 2022 @ 18:10:56.975
logdate [2022.12.10-11.57.36:345]
tags _dateparsefailure
The timestamp, logdate are different and _dateparsefailure error occurs.
In the raw logs, it starts with
[2022.12.10-11.57.36:345]
this format.
Right now,
logdate : raw log's timestamp
@timestamp : the time that log send to opensearch
I want to match logdate and @timestamp. How can I modify the filter.date.match part to make the results of the logdate and @timestamp filters the same?
If you have multiple times you can have more than one filter.date.match
, you can do this:
filter{
date {
match => ["logdate", "yyyy.MM.dd-HH.mm.ss:SSS"]
timezone => "UTC"
target=>"logdate"
}
date {
match => ["@timestamp", "yyyy.MM.dd-HH.mm.ss:SSS"]
timezone => "UTC"
target=>"@timestamp"
}
}
If your time field has multiple formats, you can do this:
date {
match => [ "logdate", "yyyy.MM.dd-HH.mm.ss:SSS", "third_format", "ISO8601" ]
target=> "@timestamp"
}
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.