How can I handle large objects and avoid an out of memory error?

Question

I'm trying to handle a large array, but the object is sometimes so large that it breaks the server with:

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

I can reproduce this locally by doing node --max-old-space-size=100 and then running my process. Without setting this option it works fine.

Here's the section of the code where it's breaking

function setManagers(json){
return Q.all(json.map(function(obj){
    return getUserByUsername(obj.username)
    .then(function(user){
        console.log('got here'); //never hits
        ...

---

function getUserByUsername(username){
    console.log('getting user'); //hits
    return UserModel.findOneQ({username:username})
    .then(function(user){
        console.log('search finished'); //never hits
        if(user){
            console.log('got user');
            return Q(user);
        }else{
            console.log('no user');
            return Q.reject({message:'user ' + username + ' could not be found locally'});
        }
    });
}

I think that this problem is due to it being a huge array, which has a length of over 5000

If I were to remove the option to limit my memory, or use a smaller data set, it'd work.

Is there any easy way to avoid this error?

Answer 1

The problem is that you're doing too many database calls simultaneously.

I would change your logic to send to the database a sorted string with all the usernames joined by commas, and I would get all the users needed at once. Then, I would loop through those users and do whatever you're doing with the users you're receiving from the database.

---

Another solution, that will not be obtrusive to your current logic, is to perform the database operations one by one, using, for example, a promise-while:

function pwhile(condition, body) {
  var done = Q.defer();

  function loop() {
    if (!condition())
      return done.resolve();
    Q.when(body(), loop, done.reject);
  }

  Q.nextTick(loop);

  return done.promise;
}

function setManagers(json) {
  var i = 0;
  return pwhile(function() { i < json.length; }, function() {
    var p = getUserByUsername(json[i].username);
    i++;
    return p;
  });
}