简体繁体 English

哪种服务器日志记录解决方案在Kafka，Fluentd和Logstash中消耗更少的处理器？

[英]Which server logging solution is less processor consuming among Kafka, Fluentd and Logstash?

原文 2016-03-29 13:26:38 6 4 logging/ logstash/ apache-kafka/ distributed/ fluentd

I have a distributed logging system to monitor load-balanced server entities. 我有一个分布式日志记录系统来监视负载平衡的服务器实体。 It is basic for me, that the server does not invest a lot of processor time in the logging process, allowing the application to run with the maximum resources possible. 对我而言，基本的做法是服务器不会在日志记录过程中花费大量的处理器时间，从而允许应用程序以最大可能的资源运行。

It would be nice to know, which of those alternatives is "cheaper" in terms of processor time or, in case, to be recommended any other solution for this matter. 很高兴知道，在处理器时间方面，哪些替代方案是“便宜的”，或者在这种情况下被推荐使用其他解决方案。

4 个解决方案

We are using Flume to collect logs and ship to Kafka - it is very low on CPU usage, and memory-wise- it is up to you how much buffering you want to do. 我们正在使用Flume收集日志并将其运送到Kafka-CPU使用率非常低，而内存方面则取决于您要执行多少缓冲。 You can also write your own Flume interceptor if you need custom parsing/routing of data. 如果需要自定义的数据解析/路由，还可以编写自己的Flume拦截器。

fluentd and logstash all have processor cost, better to not run them on your application server. 流利和logstash都有处理器成本，最好不要在应用程序服务器上运行它们。 For, kafka, since it is only its client be running on your server, it should be cheaper, but then the stability of your server depends on the stability of the kafka servers. 对于kafka，由于只有它的客户端在服务器上运行，因此它应该便宜一些，但是服务器的稳定性取决于kafka服务器的稳定性。

A better solution might be to log to files and install a cheaper log collector/forwarder to forward the log files to another server for running any log parsers with. 更好的解决方案可能是登录到文件并安装便宜的日志收集器/转发器，以将日志文件转发到另一台服务器，以运行任何日志解析器。

You should probably use just a simple lightweight rsyslog , syslog-ng or syslogd . 您可能应该只使用简单的轻量级rsyslog ， syslog-ng或syslogd 。

Depends on which technologies you'd like to use? 取决于您要使用哪种技术？

First of all, Kafka is not a log collector. 首先，Kafka 不是日志收集器。 It's a distributed message queue and can work with log collectors like Fluentd and Logstash both as consumers and producers. 它是一个分布式消息队列，可以作为使用者和生产者使用Fluentd和Logstash等日志收集器。

Instead of opinions, let's put some numbers. 让我们输入一些数字来代替意见。

CPU Usage: This depends on how much filtering and data processing you do on the client-side using Fluentd and/or Logstash. CPU使用率：这取决于您使用Fluentd和/或Logstash在客户端进行多少筛选和数据处理。 If you do minimal processing, both can process 10,000+ messages per second. 如果您进行最少的处理，则两者都可以每秒处理10,000+条消息。 Both can take advantage of multiple CPUs ( http://docs.fluentd.org/articles/in_multiprocess , for example) 两者都可以利用多个CPU（例如， http://docs.fluentd.org/articles/in_multiprocess ）
Memory: Fluentd uses about 40MB of memory and Logstash uses about 100MB of memory. 内存：Fluentd使用大约40MB的内存，Logstash使用大约100MB的内存。 If this is too much, Logstash has Beats and Fluentd as Fluentd Forwarders ( https://github.com/fluent/fluentd-forwarder ). 如果太多，Logstash将Beats和Fluentd用作Fluentd转发器（ https://github.com/fluent/fluentd-forwarder ）。