简体繁体 English

更新，添加或删除数据后，如何使用ES + Logstash + MySQL进行实时索引更新？

[英]How can I reach real-time index updating using ES + Logstash + MySQL when data was updated, added or deleted?

原文 2017-09-07 11:47:14 0 2 mysql/ elasticsearch/ indexing/ logstash

I have MySQL DB with one table without joins with ~100 millions of rows . 我有一个只有一个表的MySQL DB，没有约1亿行的联接。 These items can be updated or removed and also new records are saved into MySQL DB periodically eg every minute. 这些项目可以更新或删除，新记录也定期（例如每分钟）保存到MySQL DB中 。 How can I reach updating indexes in Elasticsearch 如何在Elasticsearch中达到更新索引

when data is updated/changed/deleted in MySQL DB?

Actualizing data is must have for me it's very important to stay up to date with MySQL. 对我来说，实现数据是必须的，与MySQL保持最新非常重要。 Is it possible? 可能吗？ Thanks. 谢谢。

Also I have tried the solution with schedule => "* * * * *" and updated_at > :sql_last_value but it is very slow. 我也尝试过使用schedule => "* * * * *" and updated_at > :sql_last_value的解决方案，但是它非常慢。

2 个解决方案

不知道您所说的“非常慢”是什么意思，但是如果它与采样频率有关，则表达式： schedule => "/2 * * * * *"将执行您在语句变量in中定义的查询.config文件每两秒钟。

Thanks for the reply to all, I've found the solution for me thanks to Aaron Mildenstein who helped me on discuss.elastic.co. 感谢所有人的答复，感谢Aaron Mildenstein帮助我在Discussion.elastic.co上找到了解决方案。 See the thread here. 在这里查看线程。

The answer: 答案：

In order to achieve this, you will need a separate process that monitors binlog transactions and initiates updates to the Elasticsearch end of things. 为了实现这一点，您将需要一个单独的过程来监视二进制日志事务并启动对Elasticsearch末尾的更新。 Elastic does not currently offer any tools that do this, so you'll have to find one that someone open sourced, or write your own. Elastic当前不提供执行此操作的任何工具，因此您必须找到别人开源的工具，或者编写自己的工具。

To parse binlogs I used this lib: https://github.com/siddontang/go-mysql-elasticsearch 为了解析二进制日志，我使用了这个库： https : //github.com/siddontang/go-mysql-elasticsearch