[英]Logstash to elasticsearch. Keys with dots
I'm facing a problem with logstash configuration. 我遇到了logstash配置问题。 You can find my logstash configuration below. 您可以在下面找到我的logstash配置。
Ruby filter removes every dot - "." Ruby过滤器删除每个点-“。” from my fields. 从我的领域。 It seems that every works fine - the result of data filtration is correct but elasticsearch magically responds with: "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Field name [/ConsumerAdminWebService/getConsumerTransactions.call] cannot contain '.'"}
where getConsumerTransactions.call
is one of my field key. 似乎每个方法都可以正常工作-数据过滤的结果是正确的,但是elasticsearch神奇地响应: "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Field name [/ConsumerAdminWebService/getConsumerTransactions.call] cannot contain '.'"}
,其中getConsumerTransactions.call
是我的字段键之一。
input {
http_poller {
urls => {
uatBackend1 => {
method => get
url => "http://some-url/"
headers => {
Accept => "application/json"
}
}
}
request_timeout => 60
# Run every 30 seconds
schedule => { cron => "* * * * * UTC"}
codec => "json"
metadata_target => "http_poller_metadata"
}
}
filter {
ruby {
init => "
def remove_dots hash
new = Hash.new
hash.each { |k,v|
if v.is_a? Hash
v = remove_dots(v)
end
new[ k.gsub('.','_') ] = v
if v.is_a? Array
v.each { |elem|
if elem.is_a? Hash
elem = remove_dots(elem)
end
new[ k.gsub('.','_') ] = elem
} unless v.nil?
end
} unless hash.nil?
return new
end
"
code => "
event.instance_variable_set(:@data,remove_dots(event.to_hash))
"
}
}
output {
elasticsearch {
hosts => localhost
}
}
I'm afraid that this line of code is not correct: event.instance_variable_set(:@data,remove_dots(event.to_hash))
- result data is somehow pinned to the event but the original data persists unchanged and is delivered to Elasticsearch api. 恐怕这行代码是不正确的: event.instance_variable_set(:@data,remove_dots(event.to_hash))
-结果数据以某种方式固定到该事件,但原始数据保持不变,并交付给Elasticsearch api。
I suppose some clarifications are required here: 我想在这里需要一些澄清:
To be honest Ruby is a magic to me :) 老实说,Ruby对我来说是一种魔力:)
If you're using the ES version 2.0 it could be a version issue where ES doesn't pick up fields which contains .
如果您使用的是ES版本2.0,则可能是版本问题,导致ES无法提取包含的字段.
dots. 点。
According to this response in this thread : 根据此线程中的此响应:
Field names cannot contain the . 字段名称不能包含。 character in Elasticsearch 2.0. 在Elasticsearch 2.0中。
As a work around you might have to mutate
(rename) your field names into something like _ or -
instead of using the .
作为一个工作,你身边可能要mutate
(重命名)的字段名弄成_ or -
而不是使用.
dot. 点。 This ticket pretty much explains this issue, where as .
这张票几乎可以解释这个问题,其中.
dots can be used in the ES versions which are after 2.0. 点可以在2.0之后的ES版本中使用。 Hope it helps! 希望能帮助到你!
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.