Node.js stopped with “Sending SIGTERM to child” for no reason

Question

This problem is pretty much the same issue posted on https://www.openshift.com/forums/openshift/nodejs-process-stopping-for-no-reason . Unfortunately it remains unanswered.

Today my Node.js app stopped few times with DEBUG: Sending SIGTERM to child... on the logfile. No more, no less. My application is a very simple single-page app with single AJAX endpoint, serving 1k-2k pageviews per day. It has been running well for days without any problem.

I use these modules:

• express
• body-parser
• request
• cheerio

-- Update:

1. I'm using one small gear. 512MB mem, 1 GB storage

2. Excerpts from log file ( ~/app-root/logs/nodejs.log )

    Thu Jul 17 2014 09:12:52 GMT-0400 (EDT) <redacted app log message> Thu Jul 17 2014 09:13:09 GMT-0400 (EDT) <redacted app log message> Thu Jul 17 2014 09:14:33 GMT-0400 (EDT) <redacted app log message> DEBUG: Sending SIGTERM to child... #### below are the log entries after issuing "ctl_app restart" DEBUG: Running node-supervisor with DEBUG: program 'server.js' DEBUG: --watch '/var/lib/openshift/redacted/app-root/data/.nodewatch' DEBUG: --ignore 'undefined' DEBUG: --extensions 'node|js|coffee' DEBUG: --exec 'node' DEBUG: Starting child process with 'node server.js' 

3. Stats from oo-cgroup-read , as suggested by @niharvey. A bit too long, so I put it on http://pastebin.com/c31gCHGZ . Apparently I use too much memory: memory.failcnt 40583 . I suppose Node.js is automatically (?) restarted on memory overusage events, but in this case it's not. I had to restart manually.

4. I forgot that I have an idle MySQL cartridge installed, now removed.

-- Update #2

The app crashed again just now. Value of memory.failcnt stays same (full stats on http://pastebin.com/LqbBVpV9 ), so it's not memory problem (?). But there are differences in the log file. The app seems restarted, but failed. After ctl_app restart it works as intented.

    Thu Jul 17 2014 22:14:46 GMT-0400 (EDT) <redacted app log message>
    Thu Jul 17 2014 22:15:03 GMT-0400 (EDT) <redacted app log message>
    DEBUG: Sending SIGTERM to child...

    ==> app-root/logs/nodejs.log-20140714113010 <==
        at Function.Module.runMain (module.js:497:10)
    DEBUG: Program node server.js exited with code 8
    DEBUG: Starting child process with 'node server.js'
    module.js:340
        throw err;
              ^
    Error: Cannot find module 'body-parser'
        at Function.Module._resolveFilename (module.js:338:15)
        at Function.Module._load (module.js:280:25)
        at Module.require (module.js:364:17)

Answer 1

To simulate this problem on your local machine run your server with supervisor in one terminal window:

supervisor server.js

Then from another terminal use the kill command

kill process_id#

The kill command with no parameters sends a SIGTERM message to the application. If supervisor receives a SIGTERM it will stop immediately.

The sample code from the sample application provided by OpenShift listens to 12 different unix signals and exits. It could be that someone at OpenShift is manually killing the process because the application is not listening to a signal that was intended to reboot it. I'm adding this code to my application to see if the behavior is more stable.

function terminator(sig){
  if (typeof sig === "string") {
    console.log('%s: Received %s - terminating sample app ...',
                 Date(Date.now()), sig);
    process.exit(1);
  }     
  console.log('%s: Node server stopped.', Date(Date.now()) );
};

process.on('exit', function() { terminator(); });

['SIGHUP', 'SIGINT', 'SIGQUIT', 'SIGILL', 'SIGTRAP', 'SIGABRT',
 'SIGBUS', 'SIGFPE', 'SIGUSR1', 'SIGSEGV', 'SIGUSR2', 'SIGTERM'
].forEach(function(element, index, array) {
  process.on(element, function() { terminator(element); });
});

Answer 2

Usually this is because your app became idle. When you ssh into the app you should see something like:

  ***  This gear has been temporarily unidled. To keep it active, access
  ***  your app @ http://abc.rhcloud.com/

You can try to use a scheduled ping to keep the app alive.

Answer 3

I had this same issue. I deleted the gear and created a new one. The new one has been running for a few days and doesn't seem to have the issue.

[Update] After a few days, the issue appeared on my new gear.