Best practices: log management for microservices based on java, running on k8s

Question

We have our java based application(s) [ we are using SpringBoot and Dropwizard ], following a microservices architecture, running on K8s clusters.

We are using log4j to print out log messages. We have configured log4j to write logs to a file. We are finding it very difficult to access these logs [ Currently we exec into the docker container and look through the logs ]. I was doing some research and many people have suggested setting log4j to write logs to the standard output stream or console rather than files. But we want the logs to be preserved for a longer duration [ more like a long time archival ].

So for this, looks like the standard is EFK or ELK [ open-source options ]. I just wanted to understand what are the best practices/configurations that one needs to follow in order to have solid logging for applications using log4j. Deploying and configuring EFK or ELK a good option? If so, could anybody let me know how I should modify my log4j properties to send logs to the stack?

Answer 1

This is an opinion question so, I'll answer in an opinionated way. We've had this issue. The problem is that log4j (or log4j2) logs are very chatty and everytime you get a Java exception, for example, it gets printed in many lines. This combined with the regular stdout logs of the container seems convoluted and would be a pain to separate all from stdout to send to Fluentd or a something like logspout

You can allocate disk space in the container and send the log4j2 logs to a file inside the container. Then you can prune your existed containers or truncate your logs on a regular schedule. With [Kubernetes](You can also read up on Kubernetes logging you can set up a Fluentd or Filebeat sidecar to ship your logs too.

But the best solution IMO, in this case, would be just to ship the log4j logs directly to an Elasticsearch cluster and possibly storing the data in its own index. This is something that you might be able to use for that.