How to list _all_ objects in Amazon S3 bucket?

Question

S3Client.ListObjects return only 1000 of objects. How to retrieve list of all existing objects using Amazon C# library?

Answer 1

As stated already, Amazon S3 indeed requires Listing Keys Using the AWS SDK for .NET :

As buckets can contain a virtually unlimited number of keys, the complete results of a list query can be extremely large. To manage large result sets, Amazon S3 uses pagination to split them into multiple responses. Each list keys response returns a page of up to 1,000 keys with an indicator indicating if the response is truncated. You send a series of list keys requests until you have received all the keys.

The mentioned indicator is the NextMarker property from the ObjectsResponse Class - its usage is illustrated in the complete example Listing Keys Using the AWS SDK for .NET , with the relevant fragment being:

static AmazonS3 client;
client = Amazon.AWSClientFactory.CreateAmazonS3Client(
                    accessKeyID, secretAccessKeyID);

ListObjectsRequest request = new ListObjectsRequest();
request.BucketName = bucketName;
do
{
   ListObjectsResponse response = client.ListObjects(request);

   // Process response.
   // ...

   // If response is truncated, set the marker to get the next 
   // set of keys.
   if (response.IsTruncated)
   {
        request.Marker = response.NextMarker;
   }
   else
   {
        request = null;
   }
} while (request != null);

Answer 2

Be aware that the answer above is not using the recommended API to List Objects: http://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html

The following snippet shows how it looks with the new API:

using (var s3Client = new AmazonS3Client(Amazon.RegionEndpoint.USEast1))
{
    ListObjectsV2Request request = new ListObjectsV2Request
    {
          BucketName = bucketName,
          MaxKeys = 10
    };
    ListObjectsV2Response response;
    do
    {
         response = await s3Client.ListObjectsV2Async(request);

         // Process response.
         // ...

         request.ContinuationToken = response.NextContinuationToken;

    } while (response.IsTruncated);        
}

Answer 3

API version has been changed; so need to do as below:

            ListObjectsV2Request request = new ListObjectsV2Request
            {
                BucketName = bucketName,
                MaxKeys = 10
            };
            ListObjectsV2Response response;
            do
            {
                response = await client.ListObjectsV2Async(request);

                // Process the response.
                foreach (S3Object entry in response.S3Objects)
                {
                    Console.WriteLine("key = {0} size = {1}",
                        entry.Key, entry.Size);
                }
                Console.WriteLine("Next Continuation Token: {0}", response.NextContinuationToken);
                request.ContinuationToken = response.NextContinuationToken;
            } while (response.IsTruncated);

For full details, please see https://docs.aws.amazon.com/AmazonS3/latest/dev/ListingObjectKeysUsingNetSDK.html

Answer 4

Use a paginator, introduced here . Amazon recommends these as a cleaner syntax compared to a do-while, and they take advantage of C# 8's IAsyncEnumerable .

IAmazonS3 S3;
CancellationToken cancellationToken;

var results = new List<S3Object>();

var paginator = S3.Paginators.ListObjectsV2(new ListObjectsV2Request
{
    BucketName = "my-bucket-name",
    Prefix = "my/prefix",
});
await foreach (var response in paginator.Responses.WithCancellation(cancellationToken))
{
    results.AddRange(response.S3Objects);
}

return results;

Answer 5

According to the documentation the client uses pagination in the case you describe. As per documentation you should use IsTruncated on the result... if it is true call ListObjects again with correctly setup Marker to get the next page etc. - stop calling when IsTruncated returns false .