获取node.js应用程序以从AWS S3存储桶中提取多个页面时遇到问题（超过1000个文件）

Question

I have inherited a node.js app and I'm trying to fake my way through fixing our issue. Everything was working great until we reached the 1000 file page limit for an AWS bucket. I have made attempts at using the .eachPage() standard and async processes and also using the .hasNextPage() process and have been unable to get it to work. I feel like of my attempts, this version is the closest to working. In my mind, it seems to make sense, but it pulls only the first 1000 files. Can someone help point me in the right direction to get those extra pages of data to load?

// Get file list from AWS s3
        s3.listObjects({
            Bucket: config.aws.s3.bucket
        }).eachPage(function(error, data, done) {
            console.log('S3 Data', data);
            done();
            if (error) {
                var err = new Error('Couldn\'t retrieve file list.');
                err.status = 404;
                return next(err);
            } else {
                return res.render('user/home', {
                    userName: user.firstName + ' ' + user.lastName,
                    dummy: (new Date()).getTime(),
                    products: util.parseProductMetaData(config.subscriptions),
                    weeks: util.parseS3FileList(user.subscriptions,
                        data.Contents,
                        config.aws.s3.worksheetFolder,
                        config.subscriptions,
                        config.aws.s3.fileRegex),
                    userHeaderContext: {
                        loggedIn: true
                    },
                    city: user.city,
                    state: user.state,
                    userFormContext: {
                        disabled: true,
                        _id: user._id,
                        email: user.email,
                        firstName: user.firstName,
                        lastName: user.lastName,
                            return val;
                        })
                    }
                });
            }
        });

Answer 1

Seems like you have some conflicting JS concepts here. When you use the asynchronous style with three parameters, as soon as you call done() the next page will execute -- since you do this right away, the next page will immediately start executing so it doesn't really make sense to return something later on.

You're also calling res.render() for each page of files, which is probably not what you intended and why it looks like only 1000 files are listed (actually you render the first 1000, then continue doing lots of work in the background after sending the result!). I would recommend something like the following:

var files = [];
s3.listObjects({
  Bucket: config.aws.s3.bucket
}).eachPage(function(error, data){
  if(err) return;
  files.push(data.Contents);
});

// Now do something with all the files
res.render('whatever',files);