[英]How to index Kafka topic key value as a field using logstash?
I would like to know to include Kafka topic key value along with the message as a separate field for indexing in Elasticsearch using Logstash.. the Kafka 'topic' weather contains the following messages as key value pairs:我想知道将 Kafka 主题键值与消息一起作为单独的字段包含在 Elasticsearch 中使用 Logstash 进行索引。Kafka“主题”天气包含以下消息作为键值对:
BOSTON:99波士顿:99
BOSTON:89波士顿:89
NYC:75纽约:75
NYC:85纽约:85
I am using the following Logstash configuration to do indexing:我正在使用以下 Logstash 配置进行索引:
input {
kafka {
bootstrap_servers => "localhost:9092"
topics => ["weather"]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "weather"
workers => 1
}
}
However, only values are getting indexed, but not the key.但是,只有值被索引,而不是键。 The index looks like this:索引如下所示:
{
"_index" : "weather",
"_type" : "doc",
"_id" : "48cGgXMBtkZlibwHwyWW",
"_score" : 1.0,
"_source" : {
"message" : "99",
"@timestamp" : "2020-07-24T13:32:50.820Z",
"@version" : "1"
}
{
"_index" : "weather",
"_type" : "doc",
"_id" : "48cGgXMBtkZlibwHwyWW",
"_score" : 1.0,
"_source" : {
"message" : "89",
"@timestamp" : "2020-07-24T13:32:50.820Z",
"@version" : "1"
}
I also need to have the city name (key of the topic) in the index.我还需要在索引中包含城市名称(主题的键)。 Once I have both, the end goal is to do different types of visualization in Kibana for each city's weather.一旦我拥有两者,最终目标是在 Kibana 中针对每个城市的天气进行不同类型的可视化。
Thanks in advance.提前致谢。
I included this in the conf file and I am now able to see the key information:我将它包含在 conf 文件中,现在我可以看到关键信息:
filter {
mutate {
copy => { "[@metadata][kafka]" => "kafka" }
}
}
input {
kafka {
bootstrap_servers => "localhost:9092"
topics => ["weather"]
decorate_events => true (!!!)
}
}
filter {
mutate {
add_field => {
"key" => %{[@metadata][kafka][key]}" (!!!)
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "weather"
workers => 1
}
}
ref: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html#_metadata_fields参考: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html#_metadata_fields
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.