在 docker 容器上缩放 filebeat

Question

I'm looking for the appropriate way to monitor applicative logs produced by nginx, tomcat, springboot embedded in docker with filebeat and ELK.

In the container strategy, a container should be used for only one purpose.

One nginx per container and one tomcat per container, meaning we can't have an additional filebeat within a nginx or tomcat container.

Over what I have read over Internet, we could have the following setup:

• a volume dedicated for storing logs
• a nginx container which mount the dedicated logs volume
• a tomcat / springboot container which mount the dedicated logs volume
• a filebeat container also mounting the dedicated logs volume

This works fine but when it comes to scale out nginx and springboot container, it is a little bit more complex for me.

Which pattern should I use to push my logs using filebeat to logstash if I have the following configuration:

• several nginx containers in load balancing with the same configuration (logs configuration is the same: same path)
• several springboot rest api containers behing nginx containers with the same configuration (logs configuration is the same:same path)

Should I create one volume by set of nginx + springboot rest api and add a filebeat container ?

Should I create a global log volume shared by all my containers and have a different log filename by container (having the name of the container in the filename of the logs?) and having only one filebeat container ?

In the second proposal, how to scale filebeat ?

Is there another way to do that ?

Many thanks for your help.

Answer 1

The easiest thing to do, if you can manage it, is to set each container process to log to its own stdout (you might be able to specify /dev/stdout or /proc/1/fd/1 as a log file). For example, the Docker Hub nginx Dockerfile specifies

RUN ln -sf /dev/stdout /var/log/nginx/access.log \
    && ln -sf /dev/stderr /var/log/nginx/error.log

so the ordinary nginx logs become the container logs. Once you do that, you can plug in the filebeat container input to read those logs and process them. You could also see them from outside the container with docker logs , they are the same logs.

---

What if you have to log to the filesystem? Or there are multiple separate log streams you want to be able to collect?

If the number of containers is variable, but you have good control over their configuration, then I'd probably set up a single global log volume as you describe and use the filebeat log input to read every log file in that directory tree.

If the number of containers is fixed, then you can set up a volume per container and mount it in each container's "usual" log storage location. Then mount all of those directories into the filebeat container. The obvious problem here is that if you do start or stop a container, you'll need to restart the log manager for the added/removed volume.

---

If you're actually on Kubernetes, there are two more possibilities. If you're trying to collect container logs out of the filesystem, you need to run a copy of filebeat on every node; a DaemonSet can manage this for you. A Kubernetes pod can also run multiple containers, so your other option is to set up pods with both an application container and a filebeat "sidecar" container that ships the logs off. Set up the pod with an emptyDir volume to hold the logs, and mount it into both containers. A template system like Helm can help you write the pod specifications without repeating the logging sidecar setup over and over.