如何处理与 nodejs 中的内存分配相关的崩溃？

Question

I am trying to fill mongoDB database with some random data from faker. So i used setInterval function to run a database Query over and over its working but the problem is when i monitor my memory usage it just growing and growing until it reaches to the that limit of v8 engine no matter how much i increase the limits it crashes anyway just a matter of time.

const faker = require("faker")
let userName; 
let email; 

setInterval(async () => {
  userName = faker.name.findName(); 
  email = faker.internet.email(); 
  await User.create({
    userName: userName,
    email: email,
  })
    .then(() => {
      //getting memory usage
      console.log(process.memoryUsage().heapUsed / 1024 / 1024);
    })
    .catch(err => {
      console.log(err);
    });
});

how should i manage memory allocation to stop crashes? is this considered to be a memory leak?

Answer 1

Calling setInterval with no time, this is adding to the callback queue as fast as possible, without any limits. And if you add items to a queue faster than you can remove them, memory will always grow.

*Edit: to understand exactly what happens with the event loop and callback queue when you do a setInterval , I recommend watching the excellent JSConf talks by Philip Roberts and/or Jake Archibald .

You have a few options. You could change to a setTimeout , and then invoke another setTimeout in your .then , once the last call has finished. This would essentially serialize it.

A more nuanced approach would be to limit the max number of concurrent mongoDB writes by using a counter. That way you can have a certain number of writes happening concurrently. In this example, it limits the number of concurrent writes to 10. This should keep your memory from growing without limits.

const faker = require("faker")
let userName; 
let email; 
let numRunning = 0;
let maxRunning = 10;

setInterval(async () => {
  if (numRunning > maxRunning) return;
  numRunning++;
  userName = faker.name.findName(); 
  email = faker.internet.email(); 
  await User.create({
    userName: createdUser.userName,
    age: createdUser.age,
  })
    .then(() => {
      numRunning--;
      //getting memory usage
      console.log(process.memoryUsage().heapUsed / 1024 / 1024);
    })
    .catch(err => {
      numRunning--;
      console.log(err);
    });
}, 50);

I also added an interval of 50ms. A zero for a setInterval really isn't a good idea. There are good reasons to sometimes use a zero for setTimeout , but I can't think of a good reason to ever use a zero for a setInterval .