Convert timestamp timezone in Logstash for output index name

Question

In my scenario, the "timestamp" of the syslog lines Logstash receives is in UTC and we use the event "timestamp" in the Elasticsearch output:

output {
    elasticsearch {
        embedded => false
        host => localhost
        port => 9200
        protocol => http
        cluster => 'elasticsearch'
        index => "syslog-%{+YYYY.MM.dd}"
    }
}

My problem is that at UTC midnight, Logstash sends log to different index before the end of the day in out timezone (GMT-4 => America/Montreal) and the index has no logs after 20h (8h PM) because of the "timestamp" being UTC.

We've done a work arround to convert the timezone but we experience a significant performance degradation:

filter {
    mutate {
        add_field => {
            # Create a new field with string value of the UTC event date
            "timestamp_zoned" => "%{@timestamp}"
        }
    }

    date {
        # Parse UTC string value and convert it to my timezone into a new field
        match => [ "timestamp_zoned", "yyyy-MM-dd HH:mm:ss Z" ]
        timezone => "America/Montreal"
        locale => "en"
        remove_field => [ "timestamp_zoned" ]
        target => "timestamp_zoned_obj"
    }

    ruby {
        # Output the zoned date to a new field
        code => "event['index_day'] = event['timestamp_zoned_obj'].strftime('%Y.%m.%d')"
        remove_field => [ "timestamp_zoned_obj" ]
    }
}

output {
    elasticsearch {
        embedded => false
        host => localhost
        port => 9200
        protocol => http
        cluster => 'elasticsearch'
        # Use of the string value
        index => "syslog-%{index_day}"
    }
}

Is there a way to optimize this config?

Answer 1

This is the optimize config, please have a try and test for the performance.

You no need to use mutate and date plugin. Use ruby plugin directly.

input {
    stdin {
    }
}

filter {
    ruby {
            code => "
                    event['index_day'] = event['@timestamp'].localtime.strftime('%Y.%m.%d')
            "
    }
}

output {
    stdout { codec => rubydebug }
}

Example output:

{
       "message" => "test",
      "@version" => "1",
    "@timestamp" => "2015-03-30T05:27:06.310Z",
          "host" => "BEN_LIM",
     "index_day" => "2015.03.29"
}

Answer 2

In logstash version 5.0 and later , you can use this:

filter{
ruby {
        code => "event.set('index_day', event.get('[@timestamp]').time.localtime.strftime('%Y%m%d'))"
    }
}

Answer 3

In version 1.5.0, we can convert timestamp by local timezone for the index name. Here is my configuration:

filter {
    ruby {
        code => "event['index_day'] = event.timestamp.time.localtime.strftime('%Y.%m.%d')"
    }
}
output {
    elasticsearch {
        host => localhost
        index => "thrall-%{index_day}"
    }
}

Answer 4

In Logstash Version 5.0.2，The API was modified. We can convert timestamp by local timezone for the index name. Here is my configuration:

filter { 
   ruby { 
       code => "event['index_day'] = event.timestamp.time.localtime.strftime('%Y.%m.%d')" 
   } 
} 

Answer 5

Similar use case - but using the logstash file output plugin and writing files dated by the local time of the arrival of the event . Verified on logstash version 7.12 .

Adapted from discuss.elastic.co , mainly zero padding the offset hours. NB! If your offset has half hours you will need to adjust accordingly.

filter {
    ruby {
        code => "
        require 'tzinfo'
        tz = 'Europe/Oslo'
        offset = TZInfo::Timezone.get(tz).current_period.utc_total_offset / (60*60)
        event.set('[@metadata][local_date]',
                  event.get('@timestamp').time.localtime(
                      sprintf('+%02i:00', offset.to_s)
                  ).strftime('%Y%m%d'))
        "
    }   
    if ([agent][type] == "filebeat") {
        mutate {
            add_field => ["file_path", "%{[host][name]}_%{[log][file][path]}.%{[@metadata][local_date]}"]
        }  
    } else {   
        mutate {
            add_field => ["file_path", "%{[agent][hostname]}_%{[agent][type]}.%{[@metadata][local_date]}"]
        }
    }
}