如何在Elasticsearch 5.x和Filebeat中解析日期

Question

I am using elasticsearch 5.x and Filebeat and want to know if there is a way of parsing date(timestamp) directly in filebeat (don't want to use logstash). I am using json.keys_under_root: true and it works great, but the problem is that timestamp (on us) is recognised as string. All of the other fields were automatically recognised as correct types only this one isn't.

How can I map it as date?

Answer 1

You can use Filebeat with the ES Ingest Node feature to parse your timestamp field and apply the value to the @timestamp field.

You would setup a simple pipeline in Elasticsearch that applies a date to incoming events.

PUT _ingest/pipeline/my-pipeline
{
  "description" : "parse timestamp and update @timestamp",
  "processors" : [
    {
      "date" : {
        "field" : "timestamp",
        "target_field" : "@timestamp"
      }
    },
    {
      "remove": {
        "field": "timestamp"
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "field": "error.message",
        "value": "{{ _ingest.on_failure_message }}"
      }
    }
  ]
}

Then in Filebeat configure the elasticsearch output to push data to your new pipeline.

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  pipeline: my-pipeline