restart node.js forever process if response time too big

Question

I got forever script for managing node.js site.

Sometimes node.js site hangs and response time go above 30 seconds. And in fact site is down. Then fast cure for it is restarting forever :

$ forever restart 3

where 3 is script number in forever list .

Is it possible to make it automatically? Is there option in forever which make it restart if response time will be more than 2 seconds for example?

Or maybe I got to run external script which will check response time and make descension to restart hanging forever script.

Or maybe I need to write this logic inside my node.js site?

Answer 1

I am assuming you want to restart the server if most of the reply are taking longer than x seconds. There are many tools that helps you to restart your instances based on their health. Monit is one of them. In this guide, monit restart the instance if reply doesn't come back in 10 seconds.

If you want to kill the instance if one request if any of the requests are taking too long, then you note down the time when you take in the request, and note down the time when request leaves. If the time is too long, throw an exception that you know will not get caught and the server would restart by itself. If you use express, then check the code for their logger under development mode, as it tracks the response time.

Answer 2

Aside leorex solution , I have something like this before to send 500 on timed out requests:

var writeHead = res.writeHead;
var timeout = setTimeout(function () {
    res.statusCode = 500;
    res.end('Response timed out');
    // To avoid errors being thrown for writes after this, they should be ignored.
    res.writeHead = res.write = res.end = function () {};
}, 40000);
res.writeHead = function () {
    // This was called in time.
    clearTimeout(timeout);
    writeHead.apply(this, arguments);
};

You can use addTimeout module to take away the timeout clearing part.

Once you implemented, you can handle as you like, you can just call process.exit(1); so forever will immediately replaces you.

You can make this smarter. Also in my application, if an uncaught error happens, I signal the supervisor process so that it will spin up another worker process and gracefully go down(close http server and wait for all pending requests to finish). You can do the same in your application, but make sure everything has a timeout callback as a failover/backup plan.