简体   繁体   中英

Logstash config output index daily by date provided by data column

My data has a column that has a recorded_date on it with the format yyyy-MM-dd HH:mm:ss . I would like to index my data daily, but I'm not sure how to append a different date format on my index name, for example measurements-yyyy-mm-dd without HH:mm:ss

input {
    file {
        path => "/measurements.txt"
        start_position => "beginning"
        sincedb_path => "/dev/null"
    }
}
filter {
    csv {
        separator => ","
        columns => ["id", "recorded_date", "unit", "Description", "CostPerUnit"]
    }

    date {
        match => [ "recorded_date", "yyyy-MM-dd HH:mm:ss" ]
        source => "@timestamp"
    }

    date {
        match => [ "recorded_date", "yyyy-MM-dd HH:mm:ss" ]
        source => "log_day"
    }

    date_formatter {
        source => "log_day"
        pattern => "YYYY-MM-dd"
    }

    mutate {convert => ["ChannelId", "integer"]}
    mutate {convert => ["NumberOfUnits", "float"]}
    mutate {convert => ["IsOpen", "integer"]}
    mutate {convert => ["CostPerUnit", "float"]}
}

output {
    elasticsearch{
        hosts => "localhost:9200"
        index => "measurements-%{log_day}"
        document_type => "measurements"
    }
    stdout {}
}
`

I should have a list of indexes with names like the following:
- measurements-2009-04-01
- measurements-2009-04-02

Why not use a grok or dissect before converting the recorded_date into date ?

With dissect, it's much easier :

dissect {
  mapping => {
    "recorded_date" => "%{log_day} %{}"
  }
}

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM