简体繁体 English

Logstash高可用性部署

[英]Logstash high availability deployment

原文 2014-07-02 15:34:40 6 3 linux/ logging/ logstash/ high-availability/ failover

I am using logstash in a mode where it reads log files from disk and puts in ElasticSearch. 我使用logstash在其从磁盘中读取日志文件，并在ElasticSearch提出的模式。

What is the best way to deploy logstash for high availability (especially failover)? 部署logstash以获得高可用性（尤其是故障转移）的最佳方法是什么？ I'm ok with both active/active mode where two logstash instances are always active, and with active/passive mode where one instance is working and the other one will start only if the first one is down. 对于两个logstash实例始终处于活动状态的主动/主动模式，以及一个实例正在工作而另一个实例仅在第一个实例关闭时才能启动的主动/被动模式，我都可以。

I'm specifically asking about logstash and not ElasticSearch. 我专门问的是logstash，而不是ElasticSearch。

3 个解决方案

It seems, that Logstash does not have built in HA options, where we remain with Linux classic - Virtual IP. Logstash似乎没有内置的HA选项，而我们仍然使用Linux经典版本-虚拟IP。 I was thinking on the same topick, and currently decided to try the following option (hot/cold version): 我在考虑相同的主题，因此决定尝试以下选项（热/冷版本）：

build 2 separate server instances with Logstash as indexer 使用Logstash作为索引器构建2个单独的服务器实例
find a way, to sync .conf files of Logstash indexer (rsync, git, etc.) 找到一种方法来同步Logstash索引器的.conf文件（rsync，git等）
use Virtual IP solution and Linux heartbeat, to move active Virtual IP between servers or use other load balancing solution, which could act as such (for example, pfsense as load balancer) 使用虚拟IP解决方案和Linux心跳，在服务器之间移动活动的虚拟IP或使用其他可以起到这种作用的负载平衡解决方案（例如pfsense作为负载平衡器）
each Logstash indexer instance has it's own REDIS instance, to keep a buffer of logs, and potentially allow to move logs from buffer, if something goes wrong with Logstash. 每个Logstash索引器实例都有其自己的REDIS实例，以保留日志缓冲区，并且如果Logstash出现问题，则有可能允许将日志从缓冲区中移出。

Here are issues, that need to be solved out, yet: 这里是需要解决的问题：

Redis can not be run in HA active/active, which introduce issue of log message routing and finding during or after instance switch. Redis无法在HA活动/活动中的HA中运行，这会在实例切换期间或之后引入日志消息路由和查找问题。
The same with Active/Passive Logstash, when switching occurs - how to get missing logs within this timeframe. 切换时，主动/被动Logstash相同-如何在此时间范围内获取丢失的日志。

As far as I know, Active/Active Logstash is available only with the following options taken in mind: 据我所知，Active / Active Logstash仅在考虑以下选项的情况下可用：

logs are doubled. 日志加倍。 If you put both indexer nodes for output in Logstash shippers. 如果将两个索引器节点都输出到Logstash运货商中。
or you have to provide logical mechanism to ship logs on conditions outside Logstash indexer configuration - eg figure it out, not to ship the same log messages to both indexers. 否则，您必须提供逻辑机制以在Logstash索引器配置之外的条件下传送日志-例如，找出原因，而不是将相同的日志消息传送到两个索引器。

You could use a queue that will act as a buffer between input and indexing process. 您可以使用一个队列，该队列将充当输入和索引过程之间的缓冲区。

It's always a good thing to separate tier with a queue, so if elasticsearch crashs, your application will not suffer. 用队列分隔层始终是一件好事，因此，如果Elasticsearch崩溃，您的应用程序将不会受到影响。

在这种情况下，最好的方法是使用某种硬件平衡器，例如F5（如果有的话）池，因此您要定义具有相应端口的VIP，然后将该VIP与主机的N个IP地址相关联。 N个logstash主机，因此您可以从logstash中获得任意数量的节点或需要运行的任何服务，然后应用循环算法并平衡连接。