List all objects in Amazon S3 using Java SDK

Question

I am listing all objects in my bucket.The question is when i am listing all objects,the last object in a batch is again considering in next batch of object and its repeating for next batch.Why is it happening as it should consider only 1000 objects in a batch and should not consider the previous batch object.

BasicAWSCredentials credentials = new BasicAWSCredentials("foo", "bar");
client = AmazonS3ClientBuilder
.standard()
.withCredentials(new AWSStaticCredentialsProvider(credentials))
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(http://(serviceEndpoint), null(signingRegion is null))
.withPathStyleAccessEnabled(true)
.withChunkedEncodingDisabled(true)
.build();
ObjectListing listing = client.listObjects( "bucketname");
System.out.println("Listing size "+listing.getObjectSummaries().size());
System.out.println("At 0 index "+ listing.getObjectSummaries().get(0).getKey());
System.out.println("At 999 index "+ listing.getObjectSummaries().get(999).getKey());
SomeFunction(listing);

while (listing.isTruncated()) {
    System.out.println("-----------------------------------------------");
    listing = client.listNextBatchOfObjects(listing);
    System.out.println("Listing size "+listing.getObjectSummaries().size());
    System.out.println("At 0 index "+ listing.getObjectSummaries().get(0).getKey());
    System.out.println("At 999 index "+ listing.getObjectSummaries().get(1000).getKey());
    someFunction(listing);
}

My ouput is:

Listing size 1000
At 0 index folder1/a.gz
At 999 index folder1/b.gz
---------------------------------------------------------------
Listing size 1001
At 0 index folder1/b.gz
At 1000 index folder1/d.gz
---------------------------------------------------------------
Listing size 1001
At 0 index folder1/d.gz
At 1000 index folder1/e.gz

As you can see the first batch 999 index is considered in second batch(same) why?It shouldn't happen right?And the next batch is taking 1001 objects including the last one from previous as it should give next 1000 not 1001.Help me solve this.Thank You.

Answer 1

It doesn't fit in the note so I put my code here:

BasicAWSCredentials credentials = new BasicAWSCredentials(accessKey, secretAccessKey);
AmazonS3 s3Client = AmazonS3ClientBuilder
        .standard()
        .withCredentials(new AWSStaticCredentialsProvider(credentials))
        .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("s3-us-west-1.amazonaws.com", region))
        .withPathStyleAccessEnabled(true)
        .withChunkedEncodingDisabled(true)
        .build();
ObjectListing listing = s3Client.listObjects(bucket);
int size = listing.getObjectSummaries().size();
if (size > 0) {
    System.out.println("Listing size " + size);
    System.out.println("At 0 index " + listing.getObjectSummaries().get(0).getKey());
    System.out.println("At " + (size - 1) + " index " + listing.getObjectSummaries().get(size - 1).getKey());
}

while(listing.isTruncated()) {
    System.out.println("-----------------------------------------------");
    listing = s3Client.listNextBatchOfObjects(listing);
    size = listing.getObjectSummaries().size();
    if (size > 0) {
        System.out.println("Listing size " + size);
        System.out.println("At 0 index " + listing.getObjectSummaries().get(0).getKey());
        System.out.println("At " + (size - 1) + " index " + listing.getObjectSummaries().get(size - 1).getKey());
    }
}

And here's the result:

Listing size 1000
At 0 index file0001.txt
At 999 index file1000.txt
-----------------------------------------------
Listing size 1000
At 0 index file1001.txt
At 999 index file2000.txt
-----------------------------------------------
Listing size 1000
At 0 index file2001.txt
At 999 index file3000.txt
-----------------------------------------------
Listing size 100
At 0 index file3001.txt
At 99 index file3100.txt

I changed your code a little on how to output sizes and indexes. By the way I'm using 1.11.330 SDK version.