要从应用服务器运送日志，该使用哪个服务器使用Logstash转发器，FLume或Fluentd？

Question

1. Logstash forwarder is light, but from logstash forwarder to logstash , there is latency over the network. [ if i am using Logstash forwarder on one machine and sending logs to Logstash which is on other machine ]

2. Flume /Flume-ng : CPU utilisation is high for same amount of data (for example for 2 MB ,its like 20 percent )

3. Fluentd : doestn't use java, its based on CRuby , but its CPU utilisation is also at peak time 30 percent, .

As per our use case we do not want to add significant load on my production boxes to just forward the log and if i use logstash i will be introducing new single point of failure so i am pretty confused to choose one among them.

Answer 1

Interesting performance statistics.

From my experience, logstash-forwarder is fairly light weight and encryption/compression is very helpful. This indeed might cause some latency. Is that an important factor for you? I guess latency is smaller than 2-3 seconds. I think that in many log management use cases, real-time is not a strong requirement.

At the end of the day, all these agents need to collect data from apps/files, package them and ship them over the network. This takes some cycles but in most cases, these are 2%-4% of the resources a normal server would have.

Have a look at rsyslog which has many configurations on how often it piggy backs logs. You can run it in a docker and limit resources more strictly on rsyslog or on any of the other agents ( https://hub.docker.com/r/logzio/logzio-rsyslog-shipper/ )

Another option would be to post logs directly from your app server with bulk HTTP post by writing your own code. It's something most open source like ELK can ingest and it something we recommend using at http://logz.io