NodeJS (express) server restarts periodically

Question

I have a simple NodeJS application that is hosted on DigitalOcean. For some reason that I cannot find, the app stops almost everyday. The time when it happens is also suspiciously very accurate, about 08:30 AM.

I have checked that the DigitalOcean host is not being restarted. Its uptime is few weeks. I have added some uncaught exception handler, hoping there is something that crashes the app but got nothing logged:

process.on('uncaughtException', function (err) {
    console.error((new Date).toUTCString() + ' uncaughtException:', err.message);
    console.error(err.stack);
    process.exit(1);
});

The server is constantly monitored by UptimeRobot and until the "crash" I can see in the log it was working fine and the last logged request was the one sent by UptimeRobot.

I have another server for different environment which is configured pretty much the same and there it is even worse, it gets down about three times a day.

And the above doesn't look like being related to server workload at all. The one that is restarted three times a day is used for some feature checks, so it is almost unused.

And now I'm stuck. I don't know how to proceed with this problem.

Below you can find the script that I use to start the server:

#!/bin/sh
set -e

### BEGIN INIT INFO
# Provides:          myapp-server
# Required-Start:    $remote_fs $syslog $network
# Required-Stop:     $remote_fs $syslog $network
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: MyApp backend server
### END INIT INFO

[ -f /etc/default/myapp-server ] && . /etc/default/myapp-server

user="myapp"
dir="/home/$user/web-server/repository/backend"
cmd="node app.js"

name="myapp-server"
pid_file="/var/run/$name.pid"
stdout_log="/var/log/$name.log"
stderr_log="/var/log/$name.err"

get_pid() {
    cat "$pid_file"
}

is_running() {
    [ -f "$pid_file" ] && ps `get_pid` > /dev/null 2>&1
}

case "$1" in
    start)
    if is_running; then
        echo "Already started"
    else
        echo "Starting $name"
        cd "$dir"
        #sudo -E -u "$user" $cmd >> "$stdout_log" 2>> "$stderr_log" &
        $cmd >> "$stdout_log" 2>> "$stderr_log" &
        echo $! > "$pid_file"
        if ! is_running; then
            echo "Unable to start, see $stdout_log and $stderr_log"
            exit 1
        fi
    fi
    ;;
    stop)
    if is_running; then
        echo -n "Stopping $name.."
        kill `get_pid`
        for i in {1..10}
        do
            if ! is_running; then
                break
            fi

            echo -n "."
            sleep 1
        done
        echo

        if is_running; then
            echo "Not stopped; may still be shutting down or shutdown may have   failed"
            exit 1
        else
            echo "Stopped"
            if [ -f "$pid_file" ]; then
                rm "$pid_file"
            fi
        fi
    else
        echo "Not running"
    fi
    ;;
    restart)
    $0 stop
    if is_running; then
        echo "Unable to stop, will not attempt to start"
        exit 1
    fi
    $0 start
    ;;
    status)
    if is_running; then
        echo "Running"
    else
        echo "Stopped"
        exit 1
    fi
    ;;
    *)
    echo "Usage: $0 {start|stop|restart|status}"
    exit 1
    ;;
esac

exit 0

And here is my package list:

{
  "name": "MyApp",
  "version": "0.0.1",
  "description": "Sample description",
  "author": "Me",
  "private": true,
  "scripts": {
    "start": "node app.js"
  },
  "engines": {
    "node": "0.10.33",
    "npm": "1.4.28"
  },
  "dependencies": {
    "bluebird": "^2.9.8",
    "bookshelf": "^0.7.9",
    "cors": "^2.7.1",
    "express": "3.4.8",
    "jade": "*",
    "knex": "^0.7.3",
    "libphonenumber": "0.0.9",
    "node-uuid": "^1.4.3",
    "node.extend": "^1.1.3",
    "nodemailer": "^1.3.4",
    "pg": "^4.2.0",
    "request": "^2.53.0",
    "ssl-root-cas": "^1.1.7",
    "xmlbuilder": "^2.6.1",
    "xoauth2": "^1.0.0"
  }
}

Update: deployed with PM2

I have deployed the app with PM2 and got nothing more from the log. A sample just before the server has crashed:

HEAD / 302 5ms - 112b
HEAD / 302 3ms - 112b
HEAD / 302 5ms - 112b
HEAD / 302 2ms - 112b
HEAD / 302 12ms - 112b
HEAD / 302 3ms - 112b
GET /tmUnblock.cgi 400 1286ms - 1.14kb
GET / 302 9ms - 62b
HEAD / 302 6ms - 112b
HEAD / 302 11ms - 112b
Environment: development
Express server listening on port 3001
HEAD / 302 20ms - 112b
HEAD / 302 10ms - 112b
HEAD / 302 10ms - 112b
HEAD / 302 2ms - 112b
HEAD / 302 3ms - 112b
HEAD / 302 2ms - 112b
HEAD / 302 7ms - 112b
HEAD / 302 2ms - 112b

Nothing suspicious, just an attempt via missing tmUnblock.cgi security holes.

Answer 1

What kind of VPS are your running RAM wise? Also what other processes are running on the server. I had an issue with Node (unrelated) where it would randomly stop working with no logging. The issue turned out to be memory usage. I'd suggest checking your memory usage with

free -m

Also I'd make sure you have a swap enabled to alleviate memory issues if this is the case. This tutorial will guide you on swap creation https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-ubuntu-14-04 assuming your on Ubuntu. A google search of how to create a swap with DO provides numerous results

Hope this helps