aws-lambda function using nodejs 12 returns error: "EMFILE, too many open files"

Question

  getCodeFromS3(s3DownloadClient, sourceBucket.key)
    .then(unzipCode)
    .then((filelist) => {
      return Promise.all([filelist, putObjects(filelist, s3UploadClient)]);
    })
    .then(putJobSuccess)
    .catch((err) => {
      putJobFailure(err);
    });

The above lambda function returns error "EMFILE, too many open files". The error is coming while unzipping the files. How can I fix it in aws console?

Answer 1

One simple way to throttle requests that consume too many file handles or sockets is to use bottleneck .

For example:

const Bottleneck = require('bottleneck');

const limiter = new Bottleneck({
  // minTime: 33,
  maxConcurrent: 100,
});

const func = async (value) => {
  // do something, returns promise
};

const values = [...];

const wrapped_func = limiter.wrap(func);
const rc = await Promise.all(values.map(wrapped_func));