使用异步js llin进行异步并行调用

Question

arrayOfFunctions is an array with 1000+ functions in it. Each function is making an async call to AWS S3 to get an object's metadata.

Function:

function (cb) {$.when(getMetadata("bucketname","objectkey"))
                    .done(function (data) {
                        cb(null, data)
                    });

For a single function, it takes less than a second to read the metadata, if I execute 1000s of them one by one that will take 1 second per function.

How I call with async.parallel:

    async.parallel(arrayOfFunctions, function (err, result) {
            console.log(result);
        }
    });

If I use async.parallel, does it mean that all the functions are executed at once and parallelly? should I get results of all of them at once and just 1-2 seconds?

thanks

Answer 1

Not quite. There's a limit on how many connections you can open at the same time. For example the web browsers place a limit on this, each browser having a different limit.

Usually, with a small # of tasks, you wouldn't hit this limit and perceive. But having 1000s of tasks trying to fetch from AWS you probably won't be having the 1-2s execution time you are expecting.

Consider also the bandwidth limit of trying to download 1000s of images at the same time, how fast your server's internet connection would have to be to download it all in 1-2s?

For further reference see this stackoverflow discussion