logstash mongodb output and ISODate type

Question

I have some troubles trying to convert a date type field into mongoDB format (ISODate).

I have a RabbitMQ queue with JSON messages inside. These messages have a Date property like this :

Date : "2014-05-01T14:53:34.25677Z"

My logstash service read the RabbitMQ queue and inject messages into mongoDB.

Here is my logstash config file :

input {
    rabbitmq  {
        ...
        codec => json
    }
}

output {

    mongodb {
        codec => json
        collection => "log"
        isodate => true
        database => "Test"
        uri => "mongodb://localhost:27017"
    }
}

My problem is that my Date property is insterted as string instead as Date. How can I do to tell Logstash to insert my Date field as an ISODate field into mongoDB?

Thank you

Answer 1

You should use a logstash Date filter to convert the string into a Date prior to inserting it into MongoDB: http://logstash.net/docs/1.4.2/filters/date

Don't know your full schema but it should looking something like this:

filter {   
  date {
    match => [ "Date", "ISO8601" ]   
    } 
}

Note the use of "ISO8601" - that appears to match the format you are receiving but you may need to play around a bit with it. As you test this I'd strongly suggest using the stdout output option for test runs to easily see what's getting done prior to insertion into MongoDB:

output {
  stdout { codec => rubydebug }
}