[英]Logstash output is incorrect
I am new to logstash and elasticsearch. 我是logstash和elasticsearch的新手。 I am using logstash to read db updates and store into elasticsearch for fast searching.
我使用logstash读取数据库更新并存储到elasticsearch中以便快速搜索。 Following is my logstash configuration file(countries.conf).
以下是我的logstash配置文件(countries.conf)。
input {
jdbc {
jdbc_driver_library => "/home/vagrant/postgresql-9.4-1201.jdbc4.jar"
jdbc_driver_class => "org.postgresql.Driver"
jdbc_connection_string => "jdbc:postgresql://192.168.10.123:5432/myDB"
jdbc_user => "myuser"
jdbc_password => "mypassword"
schedule => "* * * * *"
statement_filepath => "/home/vagrant/countries.sql"
last_run_metadata_path => "/home/vagrant/logstash/countries.log"
}
}
output {
elasticsearch {
index => "myIndex"
document_type => "countries"
document_id => "%{id}"
hosts => "localhost:9200"
}
stdout { codec => json_lines }
}
And My countries.sql file is as follows 而我的countries.sql文件如下
SELECT json.id as id,
row_to_json(json.*) AS _source
FROM (
SELECT id, created, modified, name, capital, iso_alpha2, iso_alpha3
FROM countries
) as json
I run config file using following command 我使用以下命令运行配置文件
sudo /opt/logstash/bin/logstash -f /home/vagrant/countries.conf
Output of above command on stdout is as follows:- 在stdout上输出以上命令如下: -
Settings: Default pipeline workers: 1
Pipeline main started
{"_id":6,"_source":{"type":"json","value":"{\"id\":6,\"created\":\"2013-02-07T10:11:00\",\"modified\":\"2016-04-29T11:15:40.329\",\"name\":\"Andorra\",\"capital\":\"Andorra la Vella\",\"iso_alpha2\":\"AD\",\"iso_alpha3\":\"AND\"}"},"@version":"1","@timestamp":"2016-05-02T10:08:00.931Z"}
As you can see in above output my json string in _source field is changed. 正如您在上面的输出中所看到的,_source字段中的json字符串已更改。 Ideally it should be like below
理想情况下应该如下所示
{"_id":6,"_source":{\"id\":6,\"created\":\"2013-02-07T10:11:00\",\"modified\":\"2016-04-29T11:15:40.329\",\"name\":\"Andorra\",\"capital\":\"Andorra la Vella\",\"iso_alpha2\":\"AD\",\"iso_alpha3\":\"AND\"},"@version":"1","@timestamp":"2016-05-02T10:08:00.931Z"}
Logstash is changing my json string. Logstash正在改变我的json字符串。 It is adding type:"json" an extra field and adding my actual json sting in value field.
它正在添加类型:“json”一个额外的字段,并在值字段中添加我的实际json sting。 I crossed check in db.
我越过了检查数据库。 My SQL query is correctly returning json string in format which i need.
我的SQL查询正确地以我需要的格式返回json字符串。
Can someone please let me know exactly what I am missing? 有人可以让我知道我错过了什么吗? or Can guide me in right direction?
或者可以指导我正确的方向?
Thanks in Advance! 提前致谢!
Check the logs in elasticsearch. 检查elasticsearch中的日志。 I think mostly as you are using
codec => json_lines
in stdout, this is the reason why type=>json
is getting appended. 我认为主要是因为你在stdout中使用
codec => json_lines
,这就是为什么要type=>json
的原因。 Elasticsearch records wont have the type field. Elasticsearch记录没有类型字段。
If the logs in elasticsearch also has type=>json, use mutate to remove the field. 如果elasticsearch中的日志也有type => json,请使用mutate删除该字段。
filter {
mutate {
remove_field => [ "type" ]
}
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.