[英]Import data from mysql to elasticsearch using logstash
I would like to import data from my MySQL database to elasticsearch with logstash. 我想从我的MySQL数据库将数据导入到带logstash的elasticsearch中。 I am already able to import a custom query result, but I am missing the point where I can define the mapping/settings of the index being created by logstash.
我已经能够导入自定义查询结果,但是我无法定义由logstash创建的索引的映射/设置。 Also, I have no idea how to import data with one to many relations.
另外,我也不知道如何导入具有一对多关系的数据。
This is my logstash.conf so far: 到目前为止,这是我的logstash.conf:
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://localhost:3306/shop"
jdbc_user => "root"
jdbc_password => ""
jdbc_driver_library => "C:\Users\curUser\Desktop\mysql-connector-java-5.1.42\mysql-connector-java-5.1.42-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
statement => "SELECT * FROM variants var"
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "search"
document_type => "variants"
document_id => "%{id}"
}
}
Note: I want to use logstash just to import data to elasticsearch. 注意:我只想使用logstash将数据导入elasticsearch。 This is for an online shop, so I have to use some analyzers on the fields
这是一家网上商店,所以我必须在现场使用一些分析仪
I've looked through the documentation of the ES plugin for LogStash - not sure if this is possible. 我浏览了LogStash的ES插件的文档-不确定是否可行。
However, you can set the index mapping before running LogStash. 但是,您可以在运行LogStash之前设置索引映射。 You can do so by putting the index mapping with cURL, or maybe there's a way to perform this HTTP request with LogStash
您可以通过使用cURL放置索引映射来实现,或者也许可以通过LogStash执行此HTTP请求
See https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html and https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html 参见https://www.elastic.co/guide/zh-CN/elasticsearch/reference/current/mapping.html和https://www.elastic.co/guide/zh-cn/logstash/current/plugins-outputs-elasticsearch.html
logstash.conf
file logstash.conf
文件中使用相同的索引名 If your table ( SELECT * FROM variants var
) has many columns and you don't want to import all of them, remove unwanted columns with a mutator: 如果您的表(
SELECT * FROM variants var
)具有许多列,并且您不想导入所有列,请使用一个变项删除不需要的列:
filter { mutate { remove_field => ["@version", "@timestamp", "column"]}}
If you want to rename a column use: 如果要重命名列,请使用:
filter { mutate { rename => { "id" => "ID" }}}
Join query can be added like you added the select query. 可以像添加选择查询一样添加联接查询。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.