如何仅从AWS S3对象获取顶层文件？

Question

I use Javascript to get the file names from AWS S3 bucket. After i get the complete list of file names from the bucket, i parse & manipulate it & list them in front-end. In this case, i want to get rid of the problem which will arise when the bucket with filled with huge amount of data (ie) i'll run out of memory when i try to manipulate with a shocking amount of data. So, i only need the file names of the very first layer.

Example :

The complete object in S3 bucket : { new_folder: {...}, some_file.png: {...} }

Here i only need the names -> new_folder, some_file.png

Below is the code which i use now :

const AWS = require('aws-sdk');
export default async function wasabiActions(dataObj) {
    var accessKeyId = '************';
    var secretAccessKey = '********************';

    var wasabiEndpoint = new AWS.Endpoint('s3.us-west-1.wasabisys.com');

    var s3 = await new AWS.S3({
        endpoint: wasabiEndpoint,
        accessKeyId: accessKeyId,
        secretAccessKey: secretAccessKey
    });

    var params = {
        Bucket: 'bucket_name',
    };

    s3.listObjectsV2(params, function(err, data) {
        if (!err) {
            var files = [];
            data.Contents.forEach(function(element) {
                files.push(element.Key.split('/').filter((name) => name.length > 0));
            });

            console.log(files);
            var parsedData = wasabiDataParser(files);
            console.log(parsedData);
        }
    });
}

Thanks in advance! :)

Answer 1

You can use pagination. Some AWS operations return results that are incomplete and require subsequent requests in order to attain the entire result set. Check here for more details. https://boto3.amazonaws.com/v1/documentation/api/latest/guide/paginators.html