AWS S3 Node.js SDK：需要一次从一个存储桶下载所有文件

Question

Currently I can iterate over all files in bucket and download one by one, using node.js sdk. I need to download all files from S3 bucket at once. then remove it from bucket.

Answer 1

This higher-level Node S3 library has a downloadDir function which syncs a remote S3 bucket with a local directory.

https://github.com/andrewrk/node-s3-client#clientdownloaddirparams

Answer 2

You can use the aws cli using the command below:

aws s3 sync s3://yourbucket .

If you have to use node then you can use exec or spwan .

const exec = require('child_process').exec;
exec('aws s3 sync s3://yourbucket .', (err, stdout, stderr) => {});

Answer 3

Let's say you have a function listFiles that returns an array of the Keys in the bucket you want to download and a function downloadFile that accepts a file Key (name) and downloads it and then deletes it.

You want to call downloadFile simultaneously on all files returned from listFiles right? Use the async library.

function listFiles(cb) {
    s3.stuff(params, cb);
}
function downlaodFile(key, cb) {
    s3.stuff(key, cb);
}
listFiles(function (err, fileKeys) {
    if (err) {
        throw err;//don't really but this is just an example
    }
    async.each(fileKeys, downloadFile, function done(err) {
        if (err) {
            throw err;
        }
    });
});