Running Node.js (on Windows R2 server): How can I reduce the amount of RAM that my grunt tasks consume?

Question

I've got 2 environments: One is a grunt environment that handles weekly processes that push into a live production database. The other is the live app itself with the production database.

I've recently made a few optimizations in Grunt to process and store data in a much better fashion, resulting in a faster app and a better overall UX. My problem is that the processing that I'm doing during weekly production is consuming a lot of memory and takes about 10 hours. I'm ok with it taking 10 hours, but my problem is that the memory usage goes up to about 95% and the live app's response time begins crawling and has little to no response until the grunt process completes.

My question is... without adding additional RAM (right now its 8GB), how can I make Grunt run this process without affecting the user experience in the live app? Thanks for any advice you can offer.

Answer 1

I think the "correct" answer here is to use another machine. You should not be running bulk processing tasks on the same machine that serves your production website for this exact reason – they tend to consume all available resources, leaving none left for your web server.

---

If—for whatever reason—another machine is not an option, then you need to diagnose why your bulk task is consuming so much memory. node-inspector and memwatch are good places to start. These will allow you to inspect the JavaScript heap to see what is in memory.

However, the JS heap is usually limited to somewhere between 1 and 1.9 GB (depending on node version), so if your bulk task is using more than that, there are a few possibilities:

1. There are multiple node child processes involved.
2. Your node process is allocating lots of memory outside the JS heap. This usually means you have lots of (and/or some very large) Buffer s that aren't getting disposed.
3. You may be using a native (C++) module that's leaking memory.