使用Logstash grok过滤Kafka JSON消息

Question

I try to filter kafka json messages only for Germany (DE). To do that I have to write a grok expression. Can anyone help me in writing a grok pattern for this json?

{"table":"ORDERS","type":"I","payload":{"ID":"28112","COUNTRY":"DE","AMT":15.36}}
{"table":"ORDERS","type":"I","payload":{"ID":"28114","COUNTRY":"US","AMT":25.75}}

Sorry, that I'm new to these technologies. Here is what my logstash.conf looks like:

input { 
  kafka {topics => [ "test" ] auto_offset_reset => "earliest" } 
} 

filter { 
  grok {
    match => { "message" => "?????????" }

  if [message] =~ "*COUNTRY*DE*" { 
    drop{}
  }
}      
}

output { file { path => "./test.txt"  } }

In the end I just wanna file with the Germany orders. Hope to get some help, thanks!

Answer 1

Do you need to use Logstash? If not, I would suggest a simple KSQL statement

CREATE STREAM GERMAN_ORDERS AS SELECT * FROM ORDERS WHERE COUNTRY='DE';

This creates a Kafka topic that is streamed from the first, and has just the data that you want on it. From the Kafka topic you can use Kafka Connect to land it to a file if you want that as part of your processing pipeline.

Read an example of using KSQL here , and try it out here