Logstash和ElasticSearch过滤器Date @timestamp问题

Question

Im trying to index some data from file to ElasticSearch by using Logstash .

If I'm not using the Date filter in order to replace the @timestamp everything works very well, but when in using the filter I do not get all the data.

I can't figure out why there is a difference between the Logstash command line and Elasticsearch in the @timestamp value.

Logstash conf

filter { 
    mutate {
        replace => {
            "type" => "dashboard_a"
        }
    }
    grok {
        match => [ "message", "%{DATESTAMP:Logdate} \[%{WORD:Severity}\] %{JAVACLASS:Class} %{GREEDYDATA:Stack}" ]
    }
    date {                        
        match => [ "Logdate", "dd-MM-yyyy hh:mm:ss,SSS" ]
    }   
}

Logstash Command line trace

{
**"@timestamp" => "2014-08-26T08:16:18.021Z",**
   "message" => "26-08-2014 11:16:18,021 [DEBUG] com.fnx.snapshot.mdb.SnapshotMDB  - SnapshotMDB Ctor is called\r",
  "@version" => "1",
      "host" => "bts10d1",
      "path" => "D:\\ElasticSearch\\logstash-1.4.2\\Dashboard_A\\Log_1\\6.log",
      "type" => "dashboard_a",
   "Logdate" => "26-08-2014 11:16:18,021",
  "Severity" => "DEBUG",
     "Class" => "com.fnx.snapshot.mdb.SnapshotMDB",
     "Stack" => " - SnapshotMDB Ctor is called\r"
  }

ElasticSearch result

{
    "_index": "logstash-2014.08.28",
    "_type": "dashboard_a",
    "_id": "-y23oNeLQs2mMbyz6oRyew",
    "_score": 1,
    "_source": {
        **"@timestamp": "2014-08-28T14:31:38.753Z",
        **"message": "15:07,565 [DEBUG] com.fnx.snapshot.mdb.SnapshotMDB  - SnapshotMDB Ctor is called\r",
        "@version": "1",
        "host": "bts10d1",
        "path": "D:\\ElasticSearch\\logstash-1.4.2\\Dashboard_A\\Log_1\\6.log",
        "type": "dashboard_a",
        "tags": ["_grokparsefailure"]
    }
}

Answer 1

Please make sure all your logs is in format!

You can see in the logstash command line trace the logs is

26-08-2014 11:16:18,021 [DEBUG] com.fnx.snapshot.mdb.SnapshotMDB - SnapshotMDB Ctor is called\\r

But, in the elastsicsearch the log is

15:07,565 [DEBUG] com.fnx.snapshot.mdb.SnapshotMDB - SnapshotMDB Ctor is called\\r",

Two logs have different time and their format are not same! The second one do not have any information about daytime, therefore it will cause the grok filter parsing error. You can go to check the origin logs. Or can you provide the origin logs sample for more discussion if all of them are in format!