[英]logstash 5.0.1: setup elasticsearch multiple indexes ouput for multiple kafka input topics
I have a logstash input setup as 我有一个logstash输入设置为
input {
kafka {
bootstrap_servers => "zookeper_address"
topics => ["topic1","topic2"]
}
}
I need to feed the topics into two different indexes in elasticsearch. 我需要在elasticsearch中将主题提供给两个不同的索引。 Can anyone help me with how the ouput should be setup for such a task. 任何人都可以帮我解决如何为这样的任务设置输出。 At this time I am only able to setup 这时我只能设置
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "my_index"
codec => "json"
document_id => "%{id}"
}
}
I need two indexes on the same elasticsearch instance say index1
and index2
which will be fed by messages coming in on topic1
and topic2
我需要在同一elasticsearch例如两个指标说, index1
和index2
将由未来在消息被送入topic1
和topic2
First, you need to add decorate_events
to your kafka
input in order to know from which topic the message is coming 首先,您需要向您的kafka
输入添加decorate_events
,以便知道消息来自哪个主题
input {
kafka {
bootstrap_servers => "zookeper_address"
topics => ["topic1","topic2"]
decorate_events => true
}
}
Then, you have two options, both involving conditional logic. 然后,您有两个选项,都涉及条件逻辑。 The first is by introducing a filter for adding the correct index name depending on the topic name. 第一种方法是引入一个过滤器,用于根据主题名称添加正确的索引名称。 For this you need to add 为此你需要添加
filter {
if [kafka][topic] == "topic1" {
mutate {
add_field => {"[@metadata][index]" => "index1"}
}
} else {
mutate {
add_field => {"[@metadata][index]" => "index2"}
}
}
# remove the field containing the decorations, unless you want them to land into ES
mutate {
remove_field => ["kafka"]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "%{[@metadata][index]}"
codec => "json"
document_id => "%{id}"
}
}
Then second option is to do the if/else directly in the output section, like this (but the additional kafka
field will land into ES): 然后第二个选项是直接在输出部分中执行if / else,就像这样(但是额外的kafka
字段将落入ES):
output {
if [@metadata][kafka][topic] == "topic1" {
elasticsearch {
hosts => ["localhost:9200"]
index => "index1"
codec => "json"
document_id => "%{id}"
}
} else {
elasticsearch {
hosts => ["localhost:9200"]
index => "index2"
codec => "json"
document_id => "%{id}"
}
}
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.