Logstash 到 Opensearch，_dateparsefailure 标记

Question

I have some problems while using logstash to opensearch.

filter{
    grok {
            patterns_dir => ["/etc/logstash/conf.d/patterns"]
            match => [ "message","%{DATE_FORM:logdate}%{LOGTYPE:logtype}:%{SPACE}%{GREEDYDATA:msgbody}" ]
    }
    date {
            match => ["logdate", "yyyy.MM.dd-HH.mm.ss:SSS"]
            timezone => "UTC"
            target=>"timestamp"
    }

    mutate {
            remove_field => ["message"]
            add_field => {
                    "file" => "%{[@metadata][s3][key]}"
            }
    }
}

This is the conf file I'm using for logstash. In the opensearch console

@timestamp : Dec 15, 2022 @ 18:10:56.975
logdate [2022.12.10-11.57.36:345]
tags _dateparsefailure

The timestamp, logdate are different and _dateparsefailure error occurs.

In the raw logs, it starts with

[2022.12.10-11.57.36:345]

this format.

Right now,

logdate : raw log's timestamp
@timestamp : the time that log send to opensearch

I want to match logdate and @timestamp. How can I modify the filter.date.match part to make the results of the logdate and @timestamp filters the same?

Answer 1

If you have multiple times you can have more than one filter.date.match , you can do this:

filter{
    date {
            match => ["logdate", "yyyy.MM.dd-HH.mm.ss:SSS"]
            timezone => "UTC"
            target=>"logdate"
    }
    date {
            match => ["@timestamp", "yyyy.MM.dd-HH.mm.ss:SSS"]
            timezone => "UTC"
            target=>"@timestamp"
    }
}

If your time field has multiple formats, you can do this:

date {
  match => [ "logdate", "yyyy.MM.dd-HH.mm.ss:SSS", "third_format", "ISO8601" ]
  target=> "@timestamp"
}

Reference: https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html#plugins-filters-date-match