Logging to elasticsearch using Logstash reliability and availability, handling missing logs?

Question

I 'm following this article on ELK: Building Logging System in Microservice Architecture with ELK Stack and Serilog .NET Core , in this architecture, serilog logs to Logstash and then logstash pushes to Elastic search, what if the elastic search is not reachable or if the logstash service is down or if the network is down or the system is down, in short where I 'm heading to, how can I ensure my logs are not lost in all the scenarios, the logs should be available offline as well, I thought of storing first in log files and then processing the logs from file to elastic search however the files will grow over a period of time and then I have to take care that there are no duplicate messages and the messages needs to be deleted as well, and most importantly there is no deadlock kind of situation in reading and writing file, could you please help me to understand whether ELK takes care and instead of logstash if I use fluend or fluentbit, are they better??

Code:

var log = new LoggerConfiguration()
         .WriteTo.Console()
         .WriteTo.Http("http://localhost:8080")
         .CreateLogger();


while (true)
{
    var customer = Customer.Generate();
    log.Information("{@customer} registered", customer);
    Thread.Sleep(1000);
}

output:

[13:56:02 INF] {"FirstName": "Lourdes", "LastName": "Kreiger", "SSNumber": "350-11-7869", "$type": "Customer"} registered
[13:56:03 INF] {"FirstName": "Desmond", "LastName": "Balistreri", "SSNumber": "929-58-1854", "$type": "Customer"} registered
...

Sending logs using ELK

Http input listening port 8080

input {
    http {
        #default host 0.0.0.0:8080
        codec => json
    }
}

# Separate the logs
filter {
    split {
        field => "events"
        target => "e"
        remove_field => "events"
    }
}

# Send the logs to Elasticsearch
output {
    elasticsearch {
        hosts => "elasticsearch:9200"
        index=>"customer-%{+xxxx.ww}"
    }
}

Answer 1

You have two main points of failure that could lead to data loss, the communication between logstash and elasticsearch and the communication between your service and logstash.

communication between logstash and elasticsearch

When sending data to elasticsearch logstash per default uses a memory queue between the input block and the filter block of your pipeline, this queue exists to store the events in the cases where logstash cannot talk with elasticsearch.

This memory queue has a fixed size of 1000 events, so it is not much help if you have a lot of events per second. You can change your pipeline to use the persisted queue , this queue will do the same thing as the in-memory queue, but it will write to a file in the logstash server and you can change the file size to store more events.

If the persisted queue fills up and elasticsearch is still down, logstash will stop accepting new events, when the queue will fill depends entirely on the size of the queue, the size of events and the rate of events, but the persisted queue is one of the things you can do to avoid data loss between logstash and elasticsearch.

communication between your service and logstash

If your service cannot communicate with logstash, then you will need to implement some logic on it to to avoid data loss. How to do that is entirely in your hands.

You could replicate the persisted queue that logstash uses and write to a file the events that weren't sent to logstash and then replay those events when logstash is back.

This will add a lot of extra things that you will need to implement by yourself.

alternatives

I would say that the best approach would be to just write your logs to a log file and use filebeat to send these logs to logstash or even directly to elasticsearch if you do not want to use any filters in logstash, filebeat can automatically retry sending the logs in the case of the output service is not reacheable and it also track what was already send or not.

Since you are using dotnet, you could use log4net to log, it will take care of the logging part and also rotate your logs when it reach some specified size.