Debug Node.js & Express App - Intermittently using 100% CPU

Question

I'm developing an app using NGinx + Node.js + Express + Firebase that simply takes input from a mobile app and stores it to Firebase, optionally uploading files to S3.

In its simplest terms, the "create" function does this

• Validates input
• Formats the input Checks if there is a file uploaded (via the multer plugin) and stores it
• If there was a file, upload to Amazon S3 and delete the source file (it's important to note I was encountering this issue before the inclusion of S3).
• Create the item by pushing into the items reference on Firebase
• Create the item for the user by pushing into the user_items reference on Firebase.

There are a few other functions that I have implemented as an API.

My trouble is coming from an intermittent spike in CPU usage , which is causing the nginx server to report a gateway timeout from the Node.js application.

Sometimes the server will fall over when performing authentication against a MongoDB instance, other times it will fall over when I'm recieving the input from the Mobile app. There doesn't seem to be any consistency between when it falls over. Sometimes it works fine for 15+ various requests (upload/login/list, etc), but sometimes it will fall over after just one request.

I have added error checking in the form of:

process.on('uncaughtException', function(err) {
  console.error(err.stack);
});

Which will throw errors if I mistype a variable for example, but when the server crashes there are no exceptions thrown. Similarly checking my logs shows me nothing. I've tried profiling the application but the output doesn't make any sense at all to me. It doesn't point to a function or plugin in particular.

I appreciate this is a long winded problem but I'd really appreciate it if you could point me in a direction for debugging this issue, it's causing me such a headache!

Answer 1

This may be a bug in the Firebase library. What version are you using?

I've been having a very similar issue that has had me frustrated for days. Node.js + Express + Firebase on Heroku. Process will run for a seemingly random time then I start getting timeout errors from Heroku without the process ever actually crashing or showing an error. Higher load doesn't seem to make it happen sooner.

I just updated from Firebase 1.0.14 to latest 1.0.19 and I think it may have fixed the problem for me. Process has been up for 2 hours now where it would only last for 5-30 min previously. More testing to do, but thought I'd share my in-progress results in case they were helpful.

Answer 2

It seems the answer was to do with the fact that my Express app was reusing one Firebase connection for every request, and for some reason this was causing the server to lock up.

My solution was to create some basic middleware that provides a new reference to the Firebase on each API request, see below:

var Middleware = {

    /*
     * Initialise Firebase Refs per connection
     */
    initFireBase: function(req, res, next) {

        console.log('Intialising Firebase for user');

        // We need a authToken
        var authToken = req.param('authToken');

        // Validate the auth token
        if(!authToken || authToken.length === 0) {
            return res.send(500, {code: 'INVALID_TOKEN', message: 'You must supply an authToken to this method.'});
        }
        else {

            // Attempt to parse the auth token
            try {
                var decodedToken = JWTSimple.decode(authToken, serverToken);
            }
            catch(e) {
                return res.send(500, {code: 'INVALID_TOKEN', message: 'Supplied token was not recognised.'});
            }

            // Bail out if the token is invalid
            if(!decodedToken) {
                return res.send(500, {code: 'INVALID_TOKEN', message: 'Supplied token was not recognised.'});
            }

            // Otherwise send the decoded token with the request
            else {
                req.auth = decodedToken.d;
            }

        }

        // Create a root reference
        var rootRef = new Firebase('my firebase url');

        // Apply the references to each request
        req.refs = {
            root: rootRef,
            user: rootRef.child('users'),
            inbox: rootRef.child('inbox')
        };

        // Carry on to the calling function
        next();

    }

};

I then simply call this middleware on my routes:

/*
 * Create a post
 */
router.all('/createPost', Middleware.initFireBase, function(req, res) {

  var refs = req.refs;
  refs.inbox.push({}) // etc

....

This middleware will soon be extended to provide Firebase.auth() on the connection to ensure that any API call made with a valid authToken would be signed to the user on Firebase's side. However for development this is acceptable.

Hopefully this helps someone.