Amazon AWS S3 bucket recursive fetch

Question

I'm using amazon s3 java library to connect to my s3 bucket and get the list of files. I need to browse through this directories recursively. While doing so, if I encounter a file, I need to download it. How to differentiate between directories and files in S3.

Here is my code snippet

AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
AmazonS3 s3Client = new AmazonS3Client(credentials);
ObjectListing objectsListing = s3Client.listObjects(new ListObjectsRequest(bucketName, prefix, null, null, Integer.MAX_VALUE));

Once I get the object listing of the upper level directory, I need to get the list of directories and files and process them

If there is a way to do it, please help me out with this

Thanks

Answer 1

S3 doesn't really have "directories." If you want to, you can kind of pretend like it does by using setDelimiter and getCommonPrefixes , but the default behavior is to list all objects in the bucket regardless of whether their keys contain '/'es.

If you're going to roll you own download code using AmazonS3Client, it's worth noting that S3 will return at most 1000 keys at a time from listObjects, even if you ask for more. You'll want to check the IsTruncated flag on the ObjectListing and if it's set loop around and send another request with a Marker set to get the next page of results.

Last but not least I'll throw in a plug for the TransferManager class, whose downloadDirectory method (with a keyPrefix of "") sounds like it might do exactly what you want with a single line of code?

Answer 2

There is no folders or directories in S3. File names like "abc/pqr/mno.pdf" is represented as mno.pdf with in folder pqr which in turn is with in abc, but actually it represents a single file with name "abc/pqr/mno.pdf".

Hence the recursive access logic should go in your application which is accessing the files. getObject method should be given full path like "abc/pqr/mno.pdf" to fetch the file mno.pdf