nodejs callback async calls maximum limit

Question

I'm new new to nodejs and want to have some clarity about the callstack. What my application does is read messages from SQS and upload asynchronously to different datasources like dynamo, mysql,stream etc... The problem here is using multiple workers we can receive in parallel 20/30 msgs at a time and will do async calls to upload.Since the operation is slower to upload and receive the callback, I doubt the event queue in nodejs will be filled with async calls while I make all upload async calls in a short time. So, I have two questions

1. What is the maximum capacity of event loop and how many async calls can lie in event loop at once.

2. How to control the concurrency and get the details of load and calculate wait time needed for next upload async call.

ps: Assuming I can read 80000 msgs in 10 mins, I can't make 80000 parallel async upload IO calls and queue them in nodejs event loop. Please help me understand this.

Answer 1

I don't know what is the maximum number of tasks in progress as far as the event loop is concerned, but what you can do is, you can control the number of tasks in progress with modules like Async (if you're using traditional callbacks), Q or Bluebird (if you're using promises) or with some combination of promise libraries if you're using ES8 async/await.

See:

• http://caolan.github.io/async/
• http://documentup.com/kriskowal/q/
• http://bluebirdjs.com/

Especially see things like parallelLimit etc:

• http://caolan.github.io/async/docs.html#parallelLimit

Now, for the event loop limits. You can test it with a code like this:

const n = 1000000;
let s = 0;
for (let i = 0; i < n; i++) {
  setTimeout(() => {
    s++;
    if (i >= n - 1) {
      console.log(s);
    }
  });
}

You can see that having millions of things on the event loop is not a problem so the event loop itself is unlikely to be a bottleneck in your case. I would be more concerned with your OS limit of open sockets or your server limit of concurrent requests that it can handle.

For that reason I would recommend to use one of the way to limit the number of concurrent requests that I have described above.