[英]how to filter kafka topics based on their names in logstash conf in the output section using if-else condition for elastic search
I am using the new version of Kafka plugin for elkstack 我正在为Elkstack使用新版本的Kafka插件
input {
kafka {
bootstrap_servers => "1XX.X.X.X:9092"
topics => ["test_all_logs","test_apiserver_logs","test_orchestrator_logs","test_credentialstore_logs","test_gfac_logs","local_api-orch_logs","__consumer_offsets,local_gfac_logs"]
auto_offset_reset => "earliest"
decorate_events => "true"
}
}
filter{
json {
source => "message"
target => "doc"
}
mutate { add_field => { "level" => "%{[doc][level]}"}}
mutate { add_field => { "logger" => "%{[doc][loggerName]}" } }
}
output {
stdout { codec => rubydebug }
elasticsearch {
if [@metadata][kafka][topic]== "test_all_logs" {
hosts => ["localhost:9200"]
index => ["test-all-logs-%{+YYYY.MM.dd}"]
}
if [@metadata][kafka][topic] == "test_apiserver_logs" {
hosts => ["localhost:9200"]
index => ["test_apiserver_logs-%{+YYYY.MM.dd}"]
}
if [@metadata][kafka][topic] == "test_orchestrator_logs" {
hosts => ["localhost:9200"]
index => ["test_orchestrator_logs-%{+YYYY.MM.dd}"]
}
if [@metadata][kafka][topic] == "test_credentialstore_logs" {
hosts => ["localhost:9200"]
index => ["test_credentialstore_logs-%{+YYYY.MM.dd}"]
}
if [@metadata][kafka][topic} == "test_gfac_logs" {
hosts => ["localhost:9200"]
index => ["test_gfac_logs-%{+YYYY.MM.dd}"]
}
}
}
I am following many links which did not had proper answer: How to write Logstash filter to filter kafka topics 我正在跟踪许多没有正确答案的链接: 如何编写Logstash过滤器以过滤kafka主题
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html#plugins-inputs-kafka-decorate_events https://www.elastic.co/guide/zh-CN/logstash/current/plugins-inputs-kafka.html#plugins-inputs-kafka-decorate_events
I got a suggestion that I should add decorate_events => "true"
. 我有个建议,我应该添加
decorate_events => "true"
。 and added @metadata attribute. 并添加了@metadata属性。
I executed the command: 我执行了命令:
/opt/logstash/bin/logstash -f airavata/logstash-airavata.conf --path.data ./airavata/
But I am getting the below error: 但是我收到以下错误:
Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 24, column 8 (byte 579) after output {\n stdout { codec => rubydebug }\n\n elasticsearch {\n if ",
:backtrace=>["/opt/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_ast'", "/opt/logstash/logstash-core/lib/logstash/compiler.rb:50:in `compile_imperative'", "/opt/logstash/logstash-core/lib/logstash/compiler.rb:54:in `compile_graph'", "/opt/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/opt/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:107:in `compile_lir'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:215:in `initialize'", "/opt/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:35:in `execute'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:335:in `block in converge_state'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:332:in `block in converge_state'", "org/jruby/RubyArray.java:1734:in `each'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:319:in `converge_state'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:166:in `block in converge_state_and_update'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:164:in `converge_state_and_update'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:90:in `execute'", "/opt/logstash/logstash-core/lib/logstash/runner.rb:362:in `block in execute'", "/opt/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
Can somebody help me with mistake I am doing, with proper syntax, not uncooked syntax? 有人可以用正确的语法而不是未经煮过的语法帮助我犯错吗?
Yes, you definitely need a decorate_events attribute to be set to true. 是的,您绝对需要将decorate_events属性设置为true。 It's a boolean value so you don't need to wrap it in quotes.
这是一个布尔值,因此您无需将其用引号引起来。 Just add this line to your Kafka input section:
只需将此行添加到您的Kafka输入部分:
decorate_events => true
Conditional statements are not allowed inside ElasticSearch output plugin. ElasticSearch输出插件中不允许使用条件语句。 Instead, wrap your whole elasticsearch block like:
相反,将您的整个elasticsearch块包装为:
if [@metadata][kafka][topic] == "topicA" {
elasticsearch {
index => "indexA"
...
}
} else {
elasticsearch {
index => "indexB"
...
}
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.