Logstash date filter no longer matching

Question

I'm working on a cluster of ELK services to deploy internally, and having ironed out my config on some initial test machines I'm now porting that over to a Chef cookbook. In order to make my cookbook more extensible I wrote the Logstash config template as a single function that reads the Chef node data and outputs the config. Some peculiarities with the quoting needed to happen to allow for the nested structure of an LS config file.

Anyhow, once I started getting data from my test machines I noticed that once again Logstash was using the timestamp that the event was recieved for @timestamp instead of the timestamp extracted from the event. I am at a loss as to why.

Below are config and event samples from my initial test machine, and the current test machine. I've pared down the filter statement to only the date section, and trimmed out all but the relevant event data.

Initial version:

filter {
    date {
        match => ["timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601"]
        target => "@timestamp"
    }
}

Parsed Event:

{
    "message": "Oct  1 05:32:07 web-01-01 postfix/smtp[12517]: 0E3E263266: to=<foo@blah.com>, relay=mta.blah.net[1.2.3.4]:25, delay=1.4, delays=0.23/0/0.11/1, dsn=2.0.0, status=sent (250 ok dirdel)",
    "@timestamp": "2014-10-01T05:32:07.000Z",
    "timestamp": "Oct  1 05:32:07",
}

Chef version:

filter {
    date {
        "match" => [
            "timestamp",
            "MMM  d HH:mm:ss",
            "MMM dd HH:mm:ss",
            "ISO8601"
        ]
        "target" => "@timestamp"
    }
}

Parsed Event:

{
    "message": "Oct 29 16:45:15 web-01-01 postfix/smtp[18596]: 05D9D63FA0: to=<foo@bla.com>, relay=mailin-01.mx.blah.com[1.2.3.4]:25, delay=1.1, delays=0.03/0/0.34/0.75, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 7B67F7000557B)",
    "@timestamp": "2014-10-30T18:41:33.660Z",
    "timestamp": "Oct 29 16:45:15",
}

Thanks in advance.

Edit:

Here's the full filter section, including the grok section:

filter {
    date {
        "match" => [
            "timestamp",
            "MMM  d HH:mm:ss",
            "MMM dd HH:mm:ss",
            "ISO8601"
        ]
        "target" => "@timestamp"
    }
    grok {
        "type" => "postfix"
        "patterns_dir" => [
            "/opt/logstash/etc/grok_patterns"
        ]
        "pattern" => [
            "%{SYSLOGBASE} %{POSTFIXSMTPDCONNECTS}",
            "%{SYSLOGBASE} %{POSTFIXSMTPDACTIONS}",
            "%{SYSLOGBASE} %{POSTFIXSMTPDTIMEOUTS}",
            "%{SYSLOGBASE} %{POSTFIXSMTPDLOGIN}",
            "%{SYSLOGBASE} %{POSTFIXSMTPDCLIENT}",
            "%{SYSLOGBASE} %{POSTFIXSMTPRELAY}",
            "%{SYSLOGBASE} %{POSTFIXSMTPCONNECT}",
            "%{SYSLOGBASE} %{POSTFIXSMTP4XX}",
            "%{SYSLOGBASE} %{POSTFIXSMTP5XX}",
            "%{SYSLOGBASE} %{POSTFIXSMTPREFUSAL}",
            "%{SYSLOGBASE} %{POSTFIXSMTPLOSTCONNECTION}",
            "%{SYSLOGBASE} %{POSTFIXSMTPTIMEOUT}",
            "%{SYSLOGBASE} %{POSTFIXBOUNCE}",
            "%{SYSLOGBASE} %{POSTFIXQMGR}",
            "%{SYSLOGBASE} %{POSTFIXCLEANUP}"
        ]
        "named_captures_only" => "true"
    }
}

Where the postfix patterns are from https://gist.github.com/jbrownsc/4694374 , but I don't imagine that they're terribly important in this case.

Answer 1

I built the following config based on yours, and it works. The only weird thing in the config is copying '@message' to 'timestamp'. This would normally be done by your grok{} (that you didn't post), though we do see that you do have a valid 'timestamp' field.

input {
        stdin{}
}

filter {
    mutate {
       add_field => [ "timestamp", "%{message}" ]
    }

    date {
        "match" => [
            "timestamp",
            "MMM  d HH:mm:ss",
            "MMM dd HH:mm:ss",
            "ISO8601"
        ]
        "target" => "@timestamp"
    }
}

output {
        stdout{ codec => rubydebug }
}

and the output, showing the correctly-set @timestamp.

{
       "message" => "Oct 29 16:45:15",
      "@version" => "1",
    "@timestamp" => "2015-10-29T23:45:15.000Z",
          "host" => "0.0.0.0",
     "timestamp" => "Oct 29 16:45:15"
}

Answer 2

This all appears to come down to the order in which the filters are specified. The grok filter is what creates the timestamp field, so if the date filter is specified first it has nothing to operate on.

Reversing the order so that grok comes before date has solved the problem, and it only took 10 whole months to figure out!