logstash service can't start because of permission

Question

I installed logstash as service under ubuntu 14 LTS however I'm getting the following error when starting the service ( sudo service logstash start ):

{:timestamp=>"2016-04-24T13:10:15.260000+0700", :message=>"The error reported is: \n  Permission denied - /opt/elk-stack/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/addressable-2.3.8/data/unicode.data"}

Here is permission on the unicode.data file:

-rwxrwxrwx 1 tuan sudo 115740 Jul 21  2015 /opt/elk-stack/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/addressable-2.3.8/data/unicode.data

When running command:

sudo bin/logstash -f logstash.conf

It worked!

My /etc/default/logstash file:

###############################
# Default settings for logstash
# /etc/default/logstash
###############################

# Override Java location
JAVACMD=/usr/bin/java

# Set a home directory
LS_HOME=/opt/elk-stack/logstash-1.5.3/tmp

# Arguments to pass to logstash agent
LS_OPTS=""
LS_HEAP_SIZE="500m"
LS_JAVA_OPTS="-Djava.io.tmpdir=$HOME"

# pidfiles aren't used for upstart; this is for sysv users.
LS_PIDFILE=/var/run/logstash.pid

# user id to be invoked as; for upstart: edit /etc/init/logstash.conf
LS_USER=root

# logstash logging
LS_LOG_FILE=/var/log/logstash.log
LS_USE_GC_LOGGING="true"

# logstash configuration directory
LS_CONF_DIR=/opt/elk-stack/logstash-1.5.3

# Open file limit; cannot be overridden in upstart
LS_OPEN_FILES=40

# Nice level
LS_NICE=19

# If this is set to 1, then when `stop` is called, if the process has
# not exited within a reasonable time, SIGKILL will be sent next.
# The default behavior is to simply log a message "program stop failed; still running"
KILL_ON_STOP_TIMEOUT=0

And here is the init script:

#!/bin/bash
#
# /etc/init.d/logstash -- startup script for LogStash.
#
### BEGIN INIT INFO
# Provides:          logstash
# Required-Start:    $all
# Required-Stop:     $all
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Starts logstash
# Description:       Starts logstash using start-stop-daemon
### END INIT INFO

set -e

NAME=logstash
DESC="Logstash Daemon"
DEFAULT=/etc/default/$NAME

if [ `id -u` -ne 0 ]; then
   echo "You need root privileges to run this script"
   exit 1
fi

. /lib/lsb/init-functions

if [ -r /etc/default/rcS ]; then
   . /etc/default/rcS
fi

# The following variables can be overwritten in $DEFAULT
PATH=/bin:/usr/bin:/sbin:/usr/sbin

# overwrite settings from default file
if [ -f "$DEFAULT" ]; then
   . "$DEFAULT"
fi

# Define other required variables
PID_FILE=${LS_PIDFILE}
DAEMON=/opt/elk-stack/logstash-1.5.3/bin/logstash
DAEMON_OPTS="agent -f ${LS_CONF_DIR}/${NAME}.conf -l ${LS_LOG_FILE} ${LS_OPTS}"

# Check DAEMON exists
if ! test -e $DAEMON; then
   log_failure_msg "Script $DAEMON doesn't exist"
   exit 1
fi

case "$1" in
   start)
      if [ -z "$DAEMON" ]; then
         log_failure_msg "no logstash script found - $DAEMON"
         exit 1
      fi

      # Check if a config file exists
      if [ ! "$(ls -A $LS_CONF_DIR/*.conf 2> /dev/null)" ]; then
         log_failure_msg "There aren't any configuration files in $LS_CONF_DIR"
         exit 1
      fi

      log_daemon_msg "Starting $DESC"

      # Parse the actual JAVACMD from the process' environment, we don't care about errors.
      JAVA=$(cat /proc/$(cat "${PID_FILE}" 2>/dev/null)/environ 2>/dev/null | grep -z ^JAVACMD= | cut -d= -f2)
      if start-stop-daemon --test --start --pidfile "$PID_FILE" \
         --user "$LS_USER" --exec "$JAVA" \
      >/dev/null; then
         # Prepare environment
         HOME="${HOME:-$LS_HOME}"
         LS_JAVA_OPTS="${LS_JAVA_OPTS} -Djava.io.tmpdir=${LS_HOME}"
         ulimit -n ${LS_OPEN_FILES}
         cd "${LS_HOME}"
         export PATH HOME JAVACMD LS_HEAP_SIZE LS_JAVA_OPTS LS_USE_GC_LOGGING

         # Start Daemon
         start-stop-daemon --start -b --user "$LS_USER" -c "$LS_USER":"$LS_GROUP" \
           -d "$LS_HOME" --nicelevel "$LS_NICE" --pidfile "$PID_FILE" --make-pidfile \
           --exec $DAEMON -- $DAEMON_OPTS

         sleep 1

         # Parse the actual JAVACMD from the process' environment, we don't care about errors.
         JAVA=$(cat /proc/$(cat "${PID_FILE}" 2>/dev/null)/environ 2>/dev/null | grep -z ^JAVACMD= | cut -d= -f2)
         if start-stop-daemon --test --start --pidfile "$PID_FILE" \
             --user "$LS_USER" --exec "$JAVA" \
         >/dev/null; then

            if [ -f "$PID_FILE" ]; then
               rm -f "$PID_FILE"
            fi

            log_end_msg 1
         else
            log_end_msg 0
         fi
      else
         log_progress_msg "(already running)"
         log_end_msg 0
      fi
   ;;
   stop)
      log_daemon_msg "Stopping $DESC"

      set +e

      if [ -f "$PID_FILE" ]; then
         start-stop-daemon --stop --pidfile "$PID_FILE" \
            --user "$LS_USER" \
            --retry=TERM/20/KILL/5 >/dev/null

         if [ $? -eq 1 ]; then
            log_progress_msg "$DESC is not running but pid file exists, cleaning up"
         elif [ $? -eq 3 ]; then
            PID="`cat $PID_FILE`"
            log_failure_msg "Failed to stop $DESC (pid $PID)"
            exit 1
         fi

         rm -f "$PID_FILE"
      else
         log_progress_msg "(not running)"
      fi

      log_end_msg 0
      set -e
   ;;
   status)
      set +e

      # Parse the actual JAVACMD from the process' environment, we don't care about errors.
      JAVA=$(cat /proc/$(cat "${PID_FILE}" 2>/dev/null)/environ 2>/dev/null | grep -z ^JAVACMD= | cut -d= -f2)
      start-stop-daemon --test --start --pidfile "$PID_FILE" \
         --user "$LS_USER" --exec "$JAVA" \
      >/dev/null 2>&1

      if [ "$?" = "0" ]; then
         if [ -f "$PID_FILE" ]; then
            log_success_msg "$DESC is not running, but pid file exists."
            exit 1
         else
            log_success_msg "$DESC is not running."
            exit 3
         fi
      else
         log_success_msg "$DESC is running with pid `cat $PID_FILE`"
      fi

      set -e
   ;;
   restart|force-reload)
      if [ -f "$PID_FILE" ]; then
         $0 stop
         sleep 1
      fi

      $0 start
   ;;
   *)
      log_success_msg "Usage: $0 {start|stop|restart|force-reload|status}"
      exit 1
   ;;
esac

exit 0

Anyone could help me out please?

Answer 1

When you run logstash via service it will try to start up using the logstash user (normally - depending on how it was installed and how the init script is configured).

I would check to make sure you have a logstash user:

id logstash

And you can also check the init script:

less /etc/init.d/logstash

Then search in the file for the line like:

LS_USER=logstash

In my example the logstash service tried to start up as the logstash user, if your user name to LS_USER is another name, then use that name instead in the chown below.

In order to fix this issue for you I would recommend changin ownership of all the files:

sudo chown -R logstash: /opt/elk-stack/logstash-1.5.3

Then try starting it up again with service.

Answer 2

Finally, I fixed it by changing the value:

LS_OPEN_FILES=40

to a bigger one. And in my case, it's 400 . The reason is that the init script run the command ulimit and it seems that 40 is not big enough for logstash!