'no such file or directory': Download multiple files from S3

Question

I have an API method that when called and passed a string of file keys, converts that to an array, and downloads them from S3. However when I call this API, I get the error Error: ENOENT: no such file or directory, open <filename here> on the server.

This is my api:

reports.get('/xxx/:fileName', async (req, res) => {

  var AWS = require('aws-sdk');

  var s3 = new AWS.S3();

  var str_array = req.params.fileName.split(','); 

  for (var i = 0; i < str_array.length; i++) {
    var filename = str_array[i].trim();
    localFileName = './' + filename;

    let file = fs.createWriteStream(localFileName);   

    s3.getObject({
      Bucket: config.reportBucket,
      Key: filename
    })
    .on('error', function (err) {
      res.end("File download failed with error " + err.message);            
    })
    .on('httpData', function (chunk) {
      file.write(chunk);
    })
    .on('httpDone', function () {
      file.end();
    })
    .send();
  }
  res.end("Files have been downloaded successfully")
});

How can I successfully download multiple files from S3 by passing my array of keys?

Answer 1

You need to be little more detailed with the requirements. I mean what is the use case like is there any common prefix of all S3 files that you want to download, is there any limitation on size etc. Below code uses java aws-s3-sdk 1.11.136. I am downloading all objects present at the location. Please note that this is making multiple calls (not suggested).

  public void downloadMultipleS3Objects(String[] objectPaths){
       Arrays.stream(objectPaths)
            .filter(objectPath ->   amazonS3.doesObjectExist("BUCKET_NAME",objectPath))
            .forEach(objectPath -> amazonS3.getObject("BUCKET_NAME", objectPath).getObjectContent())})};// todo you may write this stream to a file 

Note : In case you have a common prefix for all objects, there are other APIs to achieve that in a better way.

Answer 2

The issue was my AWS file keys were messing with the path for the files. AWS keys follow folder hierarchy, so files placed in a folder will have a key of folderName/fileName . I stripped the folder name off by doing this:

localFileName = './temp/' + filename.substring(filename.indexOf("/") + 1);

Also, I had to do the following to have new files be created on disk as they are being downloaded:

file = fs.createWriteStream(localFileName, {flags: 'a', encoding: 'utf-8',mode: 0666});
file.on('error', function(e) { console.error(e); });