I try to filter kafka json messages only for Germany (DE). To do that I have to write a grok expression. Can anyone help me in writing a grok pattern for this json?
{"table":"ORDERS","type":"I","payload":{"ID":"28112","COUNTRY":"DE","AMT":15.36}}
{"table":"ORDERS","type":"I","payload":{"ID":"28114","COUNTRY":"US","AMT":25.75}}
Sorry, that I'm new to these technologies. Here is what my logstash.conf looks like:
input {
kafka {topics => [ "test" ] auto_offset_reset => "earliest" }
}
filter {
grok {
match => { "message" => "?????????" }
if [message] =~ "*COUNTRY*DE*" {
drop{}
}
}
}
output { file { path => "./test.txt" } }
In the end I just wanna file with the Germany orders. Hope to get some help, thanks!
Do you need to use Logstash? If not, I would suggest a simple KSQL statement
CREATE STREAM GERMAN_ORDERS AS SELECT * FROM ORDERS WHERE COUNTRY='DE';
This creates a Kafka topic that is streamed from the first, and has just the data that you want on it. From the Kafka topic you can use Kafka Connect to land it to a file if you want that as part of your processing pipeline.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.