Right tools to implement logging and monitoring

Question

My team and I recently started working on a new project. Our project is layered in 3 tiers - client (Winforms Application), middle ware (.NET Core 3.0 Web API), Database (PostgreSQL). I was assigned to find the optimal way to implement a logging and monitoring system.

What we need is log every exception (of course), as well as every query that our middle ware does to our database and also some audit logs. Bear in mind that these logs will occur very often since the users are more than 200 concurrently.

At first my search began from the noSQL databases. I was thinking of picking MongoDb, Cassandra or ElasticSearch, since they are fast and do not need a specific schema and log everything there. But then I stumbled upon this great blog post by Nick Craver - the Architecture Lead for Stack Exchange. There he mentions the use of StackExchange.Exceptional and I think that this is what we actually need. But I am not sure how to actually use it and which of all those great solutions that Nick gives I need for my use case.

So I am asking:

1. if you have ever used Exceptional enlighten me of its uses
2. do i need OPServer too to monitor the logs?
3. am I still better using just Mongo?

Answer 1

If you asking me I will be going with ELKB stack. I haven't use OPServer. So I am not going to comment on this. But I would like to share my opinion about elasticsearch how your use case can be achievable on this.

As far I understand you have two things need to log:

1. Every Exception/Errors
2. Middleware Query logs

ELKB stack is designed for these use cases only. Where there will be a river kind of flow. Your logs will keep storing in Elasticsearch.

Below are a few steps for how you can achieve your use case:

Step 1 : Setup Elasticsearch. For this, you don't need to be any professional. Just follow the wonderfull document they have.
Step 2 : Setup Logstash / Filebeat which will read your logs file in realtime and keep inserting in Elasticsearch.
Step 3 : Setup Kibana over the elasticsearch. You will get a complete view of your data logs.
Step 4 : Just add logs point in your app where you will get Exceptions or you want to log something else.

PROS :

1. All tools are readymade and used by many other organizations with very heavy use cases. So it will be fit in your use case too.
2. Very good documentation and community support.
3. Great data view by kibana. You can create your own dashboard. It is like one view for your whole system logs.
4. There are already Some of beats available like MySQL beats, Nginx Beats, Apache beats, Mongo DB, etc.. Here you have to just plug and start using those.
5. It will be your centralize architecture whenever you want to add some new type of logs, You need to just start writing in the same log file and it will start logging.

There are lots of prons you will find on the Internet.

CONS :

1. You can set alert over this using 'watcher', which is paid.
2. If you want to set open source alert, You can go with alertmanager (Prometheus) where you need to do some extra integrations.

Below are a few Use cases I would like to share. I am sure it will help you out to understand:

Dream11 : https://medium.com/@D11Engg/elasticsearch-dream11-30328d913cd5
Uber : https://eng.uber.com/elk/
Ebay : https://www.elastic.co/blog/monitoring-petabytes-of-logs-at-ebay-with-beats