用logstash解析动态红宝石哈希

Question

I use logstash and I have Ruby hashes in my log. Logs looks like:

 id: 20171023080217469299836 time: 2017-10-23 08:02:17 +0500 login: 123 params: {:service_id=>21164, :user_id=>"771713"}
 id: 20171022185107064615881 time: 2017-10-22 18:51:07 +0500 login: 321 params: {:unc=>"521130929", :id=>"107005094"}

And I parse it into field "params"

   id: 20171023080217469299836 time: 2017-10-23 08:02:17 +0500 login: 123 params: {:service_id=>21164, :user_id=>"771713"}                         
   {                                                                                                                                               
       "@timestamp" => 2017-10-23T03:02:17.000Z,                                                                                                   
         "@version" => "1",                                                                                                                        
             "host" => "elk",                                                                                                                
               "id" => "20171023080217469299836",                                                                                                 
            "login" => "123",                                                                                                                     
           "params" => "{:service_id=>21164, :user_id=>\"771713\"}",                                                                                                                                                                                                                     
   }                                                                                                                                               
   id: 20171022185107064615881 time: 2017-10-22 18:51:07 +0500 login: 321 params: {:unc=>"521130929", :id=>"107005094"}                            
   {                                                                                                                                               
       "@timestamp" => 2017-10-22T13:51:07.000Z,    
         "@version" => "1",                                                                                                                        
             "host" => "elk",                                                                                                                
               "id" => "20171022185107064615881",                                                                                                 
            "login" => "321",                                                                                                                     
           "params" => "{:unc=>\"521130929\", :id=>\"107005094\"}",                                                                                                                                                                                                                         
   }                                                                                                                                               

Parameter names various. and can also be added or deleted (there are sometimes 2 or 4-5 parameters in hash). I'd like to parse this into different fields (like xml-filter parses)

{
"parse.service_id" : 21164, 
"parse.user_id" : 771713
}

and

{
"parse.unc" : 521130929 
"parse.id" : 107005094
}

But can't find how to do this. Has Logstash ruby parser?

Answer 1

You can use the kv filter and the grok filter together.

In your filter.conf logtash file, add the following:

grok {
  match => { "message" => "id: %{INT:id} time: %{GREEDYDATA:time} login: %{INT:login} params: %{GREEDYDATA:params}" }
}
date {
    match => ["time", "yyyy-MM-dd HH:mm:ss Z"]
  }
kv {
  source => "params"
  remove_char_key => ":"
  remove_char_value => "\""
  field_split => ","
  value_split => "="
  trim_key => "\{"
  trim_value => "\}>\""
  prefix => "parse."

}

PS: you need to escape special regex characters like " and { using \\ .

This will give you the following visualization: