Extracting a substring after a match position using grok in logstash

Question

Objective : I have a log file from where I want to extract the amount details after the string Amount::: in the below given log file.

What I have Done so far: Since it is a Custom Parsing, I have created a custom pattern using RegEx and I am trying to Implement it using logstash.

here is my log file -

28-04-2017 14:45:50 INFO  abcinfo (ABC_TxnLog_ServiceImpl.java295) - Amount::: 3000.00  
28-04-2017 12:45:50 INFO  abcinfo (ABC_TxnLog_ServiceImpl.java295) - Amount::: 31000.00  
28-04-2017 14:15:50 INFO  abcinfo (ABC_TxnLog_ServiceImpl.java295) - Amount::: 10000.00  
28-04-2017 11:45:50 INFO  abcinfo (ABC_TxnLog_ServiceImpl.java295) - Amount::: 9000.00  
28-04-2017 08:15:50 INFO  abcinfo (ABC_TxnLog_ServiceImpl.java295) - Amount::: 7000.00

I have used Regex to find the string Amount:::
Note : I want to extract the sub string which is coming after the string Amount:::

here are my Custom Patterns I have used in Grok:
(but it doesn't yield good results)

CUSTOM_AMOUNT (?<= - Amount::: ).*    
CUSTOM_AMOUNT (?<=Amount::: )%{BASE16FLOAT}

here is my logstacsh.conf -

input { 
    file {
       path => "D:\elk\data\amnt_parse.txt"
       type => "customgrok"
       start_position => "beginning"
       sincedb_path => "/dev/null"
         } 
      }  
 filter{ 
       if[type]== "customgrok" {

            if "_grokparsefailure" in [tags] { 
                              grok { 
                                   patterns_dir => "D:\elk\logstash-5.2.1\vendor\bundle\jruby\1.9\gems\logstash-patterns-core-4.0.2\patterns\custom" 

                                    match => { "message" => "%{CUSTOM_AMOUNT:amount" } 
                                    add_field => { "subType" => "Amount"           } 

    } 

    }
    }  
 mutate {
      gsub => ['message', "\t", " "] 
        }  } }

 output {
     stdout {
         codec => "rubydebug"
            }
     elasticsearch {
         index => "amnt_parsing_change"
          hosts =>"localhost"

            }
            }   

Our intension is to Visualize and to perform aggregation operations based on the extracted substring using Kibana and Elasticsearch.
but it stores the log file into the variable "message" . as you can see here, match => { "message" => "%{CUSTOM_AMOUNT:amount" } .

here is how the line is stored inside "message" , when I tried to view it in Kibana -

"message": "28-04-2017 11:45:50 INFO  abcinfo (ABC_TxnLog_ServiceImpl.java295) - Amount::: 9000.00\r",  
"message": "28-04-2017 12:45:50 INFO  abcinfo (ABC_TxnLog_ServiceImpl.java295) - Amount::: 31000.00\r",    
"message": "28-04-2017 11:45:50 INFO  abcinfo (ABC_TxnLog_ServiceImpl.java295) - Amount::: 9000.00\r",  

Logstash file is loading the Data(log file) and Index is also getting created but Custom Pattern isn't giving expected result. what are possibilities to extract the sub string which I have mentioned above ? or do we have any alternatives?

Answer 1

Here is what you have to do :

filter {
     grok {
                match => {
                        "message" => "%{DATESTAMP:Date} %{WORD:LogSeverity}\s+%{WORD:LogInfo} \(%{NOTSPACE:JavaClass}\) \- Amount::: %{NUMBER:Amount}"
                        }
        }
                mutate
                        {
                                gsub =>
                                [
                                        "Data"," ","-"
                                ]
                                #If you dont want those fields
                                remove_field => ["Date","LogSeverity","LogInfo","JavaClass"]

                        }
        }

I recommend you to read the documentations :

Grok Documentation Grok Patterns

You can use the following debugger :

GrokDebbuger