Docker container Application logs to ELK stack without filebeat

Question

I'm using the Elasti Cloud as it appears to be the most suitable for quickly setting up application logging. I have 24 docker container running in different nodes, and some containers have no of replicas also. i want to export inside docker container logs to elk stack.. I don't want to install Filebeat on each of my containers because that seems like it goes directly against Docker's separation of duties mantra.

.... how do I get logs from my application containers to log stash server

Answer 1

You can send your syslog to Logstash by configuring rsyslogd like this

# /etc/rsyslog.d/99-ship-syslog.conf
*.*;syslog;auth,authpriv.none action(
  type="omfwd"
  Target="myremote.elk-server.net"
  Port="5001"
  Protocol="udp"
)

If you don't have rsyslog running yet, you can add it like so (alpine linux example):

# Dockerfile
FROM alpine:3.7

RUN  apk update \
  && apk add rsyslog

COPY rsyslog.conf /etc/rsyslog.conf

EXPOSE 514 514/udp

VOLUME [ "/var/log", "/etc/rsyslog.d" ]
ENTRYPOINT [ "rsyslogd", "-n" ]

--

# rsyslogd.conf
#
# if you experience problems, check:
# http://www.rsyslog.com/troubleshoot

#### MODULES ####

module(load="imuxsock")    # local system logging support (e.g. via logger command)
#module(load="imklog")     # kernel logging support (previously done by rklogd)
module(load="immark")      # --MARK-- message support
module(load="imudp")       # UDP listener support


input(type="imudp" port="514")

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 action(type="omfile" file="/dev/console")

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                action(type="omfile" file="/var/log/messages")

# The authpriv file has restricted access.
authpriv.*                                              action(type="omfile" file="/var/log/secure")

# Log all the mail messages in one place.
mail.*                                                  action(type="omfile" file="/var/log/maillog")

# Log cron stuff
cron.*                                                  action(type="omfile" file="/var/log/cron")

# Everybody gets emergency messages
*.emerg                                                 action(type="omusrmsg" users="*")

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          action(type="omfile" file="/var/log/spooler")

# Save boot messages also to boot.log
local7.*                                                action(type="omfile" file="/var/log/boot.log")

# log every host in its own directory
if $fromhost-ip then /var/log/$fromhost-ip/messages

# Include all .conf files in /etc/rsyslog.d
$IncludeConfig /etc/rsyslog.d/*.conf
$template GRAYLOGRFC5424,"<%PRI%>%PROTOCOL-VERSION% %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n"
*.info;mail.none;authpriv.none;cron.none;*.* @@graylog:514;GRAYLOGRFC5424 # forward everything to remote server

As you're running within a java-application, you can even send you logs directly to syslog. Here's a small configuration example with log4j

log4j.rootLogger=INFO, SYSLOG

log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
log4j.appender.SYSLOG.syslogHost=myremote.elk-server.net
log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout
log4j.appender.SYSLOG.layout.conversionPattern=%d{ISO8601} %-5p [%t] %c{2} %x - %m%n
log4j.appender.SYSLOG.Facility=LOCAL1