Setting up logs location for Docker Logstash image

Question

Am experimenting with creating a Docker image for Logstash interacting with a custom http . Wondering what would be a way to set up the logistics for the logs locations which Logstash will be reading off of. The Logstash process runs on EC2 and will be reading from Apache, HAProxy, Jboss applications utilizing File based inputs. Would highly appreciate if anyone can throw more light on this.

Answer 1

The log files generated by Apache, HAProxy, Jboss and anything else you want to pick up with your Logstash Docker container should be left where they would normally be located, and and specifically referenced in the docker run command. You gain:

1. Not having to change your current services configuration.
2. Not having to remember/locate your log files on the host in case something goes wrong and you need direct access to the files themselves. They will be in the same place your are used to.

What you need to do is map the logs files from where they are, to the running docker container. You could do that to the same directory inside the container, or to separate directories, it is up to you. If you don't have any filename collisions, I would map all the files to the same directory inside the container. An example would be something like this:

docker run -v /var/log/haproxy/error.log:/host/logs/haproxy_error.log \
           -v /var/log/httpd/access_log:/host/logs/apache_access.log \
           -v /var/log/some_other_log.log:/host/logs/some_other_log.log \
my-logstash-docker-image:latest

You will have to make sure that your Logstash configuration inside the Docker container looks for the files in the following locations:

• /host/logs/haproxy_error.log
• /host/logs/apache_access.log
• /host/logs/some_other_log.log

You can check out how I did something similar to this mapping for the Logz.io log shipper Docker image.