如何使用 Nodejs 在两个 S3 存储桶之间复制对象

Question

I've to copy S3 objects (text and media files) from one S3 bucket into another. Although bucket names are different, both buckets allow access using the same credentials. How can I achieve that using Nodejs?

I've so far been able to get a list of objects. For some reason it just gives me the list of first 100 objects and show that 900 more items are available. Here is what I've so far:

index.js

require("dotenv").config();
const AWS = require('aws-sdk');

let s3 = new AWS.S3({
    secretAccessKey: process.env.SECRET_ACCESS_KEY,
    accessKeyId: process.env.ACCESS_KEY_ID,
    region: process.env.REGION
});

const BUCKET = process.env.BUCKET_NAME;
console.log(BUCKET);

var params = {
    Bucket: BUCKET, 
};

const obj = s3.listObjectsV2(params, function(err, data) {
    if (err) console.log(err, err.stack); // an error occurred
    else     console.log(data);           // successful response
})

I've two questions:

1. How can I copy S3 objects between S3 buckets? Do I need to use a combination of getObject() and putObject() or copyObjects() ?
2. Why the listObject() only gives me 100 objects with a mention of 900 more items. The total number of S3 objects is a lot more than these numbers combined. How can I get all the objects in the bucket?

Answer 1

You can use copyObject to move an object between buckets. Just make sure you pass CopySource as the object you want to copy, and Bucket as the target bucket.

To answer your second question: listObjectsV2 is a paginated SDK function. If you want to list more than the max (defaults to 1,000) objects, you'll have to iteratively call listObjectsV2 with the StartAfter option set to the last key of the previous request, which will start listing objects after that key. Keep calling this, updating StartAfter , until some number of objects is returned less than 1,000.