I don't know how to filter my log file with grok and logstash

Question

I have an small java app that loads logs similar to these ones bellow:

Fri May 29 12:10:34 BST 2015 Trade ID: 2 status is :received
Fri May 29 14:12:36 BST 2015 Trade ID: 4 status is :received
Fri May 29 17:15:39 BST 2015 Trade ID: 3 status is :received
Fri May 29 21:19:43 BST 2015 Trade ID: 3 status is :Parsed
Sat May 30 02:24:48 BST 2015 Trade ID: 8 status is :received
Sat May 30 08:30:54 BST 2015 Trade ID: 3 status is :Data not found
Sat May 30 15:38:01 BST 2015 Trade ID: 3 status is :Book not found
Sat May 30 23:46:09 BST 2015 Trade ID: 6 status is :received

I want to use ELK stack to analyse my logs and filter them. I would like at least 3 filters : Date and time, trade Id and status.

In the filter part of my logstash configuration file here is what I did:

filter {
grok {
    match => { "message" => "%{DAY} %{MONTH} %{DAY} %{TIME} BST %{YEAR} Trade ID: %{NUMBER:tradeId}  status is : %{WORD:status}" }
  }

And for the moment I can't filter my logs as I want.

Answer 1

You have some extra spaces between the pattern, and for the status, you would like to parse the entire message, so using the GREEEDYDATA instead of the WORD is your choice.

filter {
    grok {
        match => { "message" => "%{DAY:day} %{MONTH:month} %{MONTHDAY:monthday} %{TIME:time} BST %{YEAR:year} Trade ID: %{NUMBER:tradeId} status is :%{GREEDYDATA:status}" }
    }
}

For this log line:

Sat May 30 15:38:01 BST 2015 Trade ID: 3 status is :Book not found

You will end up with a json like:

{
   "message" => "Sat May 30 15:38:01 BST 2015 Trade ID: 3 status is :Book not found",
  "@version" => "1",
"@timestamp" => "2015-08-18T18:28:47.195Z",
      "host" => "Gabriels-MacBook-Pro.local",
       "day" => "Sat",
     "month" => "May",
  "monthday" => "30",
      "time" => "15:38:01",
      "year" => "2015",
   "tradeId" => "3",
    "status" => "Book not found"

}