使用Java SDK列出AWS S3中的所有對象

Question

我正在使用以下代碼段列出存儲桶中的對象。

objectListing = client.listObjects(bucketname);
do{
   for (S3ObjectSummary objectSummary : objectListing.getObjectSummaries()) {
         System.out.printf(" - %s (size: %d)\n", objectSummary.getKey(), objectSummary.getSize());
        }
         objectListing=s3Client.listNextBatchOfObjects(objectListing);
  }while (objectListing.isTruncated());

我無法獲取最后一批對象。我對此進行了一些研究，並將這些批次保存在列表中。但是由於有上百萬個對象，因此無法使用列表來保存所有對象，這將導致有時是堆內存問題。我該如何獲取所有對象。謝謝！！！

新：

我正在運行此：

    BasicAWSCredentials credentials = new BasicAWSCredentials("foo", "bar");
    client = AmazonS3ClientBuilder
    .standard()
    .withCredentials(new AWSStaticCredentialsProvider(credentials))
    .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("http://localhost:" + port, null))
    .withPathStyleAccessEnabled(true)
    .withChunkedEncodingDisabled(true)
    .build();
    ObjectListing listing = client.listObjects( "bucketname");
    System.out.println("Listing size "+listing.getObjectSummaries().size());
    System.out.println("At 0 index "+ listing.getObjectSummaries().get(0).getKey());
    System.out.println("At 999 index "+ listing.getObjectSummaries().get(999).getKey());
    while (listing.isTruncated()) {
              System.out.println("-----------------------------------------------");
        listing = client.listNextBatchOfObjects(listing);
        System.out.println("Listing size "+listing.getObjectSummaries().size());
        System.out.println("At 0 index "+ listing.getObjectSummaries().get(0).getKey());
        System.out.println("At 999 index "+ listing.getObjectSummaries().get(1000).getKey());
}

我得到以下結果：

Listing size 1000
At 0 index folder1/a.gz
At 999 index folder1/b.gz
---------------------------------------------------------------
Listing size 1001
At 0 index folder1/b.gz
At 1000 index folder1/d.gz
---------------------------------------------------------------
Listing size 1001
At 0 index folder1/d.gz
At 1000 index folder1/e.gz

Answer 1

簡單明了

ObjectListing listing = s3.listObjects( bucketName, prefix );
List<S3ObjectSummary> summaries = listing.getObjectSummaries();

while (listing.isTruncated()) {
   listing = s3.listNextBatchOfObjects (listing);
   summaries.addAll (listing.getObjectSummaries());
}  

要么

ObjectListing listing = s3.listObjects( bucketName, prefix );
doSomeProcessing(listing);

while (listing.isTruncated()) {
   listing = s3.listNextBatchOfObjects (listing);
   doSomeProcessing(listing);
}  

更新：
關於重復元素的以下注釋，我運行了以下代碼

是的，我正在獲取對象，但是重復了1000和1001對象，因此重復了2001和2002對象，依此類推。如何通過第二種方法@raevilman避免這種情況。 謝謝

public static void main(String[] args) {
        int i=0;
        System.out.println("start");
        ObjectListing listing = s3Client.listObjects( "emr-logs");
        System.out.println("Listing size "+listing.getObjectSummaries().size());
        System.out.println("At 0 index "+ listing.getObjectSummaries().get(0).getKey());
        System.out.println("At 999 index "+ listing.getObjectSummaries().get(999).getKey());
        while (listing.isTruncated()) {
            if(i>3)break;
            System.out.println("========================================================================");
            listing = s3Client.listNextBatchOfObjects(listing);
            System.out.println("Listing size "+listing.getObjectSummaries().size());
            System.out.println("At 0 index "+ listing.getObjectSummaries().get(0).getKey());
            System.out.println("At 999 index "+ listing.getObjectSummaries().get(999).getKey());
            i++;
        }
        System.out.println("end");
    }

我得到了以下結果，沒有重復元素

start
Listing size 1000
At 0 index j-10HD9DMBVVTJL/containers/application_1507189355052_0001/container_1507189355052_0001_01_000001/stderr.gz
At 999 index j-156WGS0LMKA2I/node/i-00085367e194fc02a/daemons/instance-state/instance-state.log-2017-11-16-05-15.gz
========================================================================
Listing size 1000
At 0 index j-156WGS0LMKA2I/node/i-00085367e194fc02a/daemons/instance-state/instance-state.log-2017-11-16-05-30.gz
At 999 index j-182UIXOOU8GZ6/node/i-061ffd1d1ae11da74/provision-node/0d1707a0-71dd-4dd5-a1dc-ab226ee2d150/stdout.gz
========================================================================
Listing size 1000
At 0 index j-182UIXOOU8GZ6/node/i-061ffd1d1ae11da74/provision-node/apps-phase/stderr.gz
At 999 index j-1BW9J554DDY15/containers/application_1521803257216_0002/container_1521803257216_0002_01_000002/stderr.gz
========================================================================
Listing size 1000
At 0 index j-1BW9J554DDY15/containers/application_1521803257216_0002/container_1521803257216_0002_01_000002/stdout.gz
At 999 index j-1EKRPTSEXCTB5/node/i-0576a3c452d00384b/applications/hadoop/steps/s-2B5LZ2PC741FD/controller.gz
========================================================================
Listing size 1000
At 0 index j-1EKRPTSEXCTB5/node/i-0576a3c452d00384b/applications/hadoop/steps/s-2B5LZ2PC741FD/stderr.gz
At 999 index j-1G6AYY5EMTR94/node/i-02363f6ac11c89135/daemons/instance-state/instance-state.log-2017-10-29-14-15.gz
end

Process finished with exit code 0