How to crawl nginx container logs via filebeat?

Question

Problem Statement

The NGINX image is configured to send the main NGINX access and error logs to the Docker log collector by default. This is done by linking them to stdout and stderr, which causes all messages from both logs to be stored in the file /var/lib/docker/containers/<container id>/<container id>-json.log on the Docker Host.

Since the hard work of getting the logs out of the container and into the host has already been taken care of us, perhaps we should try to leverage that? But there are numerous indistinguishable folders in /var/lib/docker/containers/

# ls -alrt /var/lib/docker/containers/
total 84
drwx--x--x 14 root root 4096 Jul  4 13:40 ..
drwx------  4 root root 4096 Jul  4 13:55 a4ee4224c3e4c68a8023eb63c01b2a288019257440b30c4efb7226eb83629956
drwx------  4 root root 4096 Jul  6 16:24 59d1465b5c42f2ce6b13747c39ff3995191d325d641b6ef8cad1a8446247ef24
...
drwx------  4 root root 4096 Jul  9 06:34 cab3407af18d778b259f54df16e60f5e5187f14b01a020b30f6c91c6f8003bdd
drwx------  4 root root 4096 Jul  9 06:35 0b99140af456b29af6fcd3956a6cdfa4c78d1e1b387654645f63b8dc4bbf049c
drwx------ 21 root root 4096 Jul  9 06:35 .

Even if we narrow them down by searching recursively through /var/lib/docker/containers/ for any files that are of type -json.log and contain the string upstream_response_time

# grep -lr "upstream_response_time" /var/lib/docker/containers/ --include "*-json.log"
/var/lib/docker/containers/cfe8...fe18/cfe8...fe18-json.log
/var/lib/docker/containers/c3c3...6662/c3c3...6662-json.log

... still leaves us in a situation where we will constantly have to step in to find the correct folders due to containers starting/stopping ... we would be stuck reconfiguring FileBeat to crawl them.

Question: So how can the docker container log folders be renamed to give them a predictable name?

Alternatives

Here are certain other methods that I've ruled out but feel free to differ.

Setting up a named volume

$ tree /var/lib/docker/volumes/*nginx-log-volume
/var/lib/docker/volumes/my_swarm_stack_nginx-log-volume
└── _data
    ├── access.log -> /dev/stdout
    └── error.log -> /dev/stderr

The named volume exists as a combination of the stack name and the named volume name: my_swarm_stack_nginx-log-volume . BUT rather than being regular files, these are some sort of a softlink/pipe to std streams. So I felt that this approach is invalid.

Answer 1

I think you are over-complicating the problem at hand. Filebeat already has a lot of configurable options, you don't need to reinvent stuff like this.

I suggest you just use add_docker_metadata processor. This will attach useful information like image & container name for each log produced by the container, which could then be checked by drop processor and you could make the conditions here such that you only accept logs from a specific container only.

processors:
- add_docker_metadata:
- drop_event:
    when:
      not:
         regexp:
            docker.container.name: "^nginx"

Adding Docker Metadata Documentation

Filtering Using Drop Processor