Amazon S3 listing "directories"

Question

I've created a hierarchy in S3 via the AWS S3 Management Console. If I run the following code to list the bucket:

AmazonS3 s3 = new AmazonS3Client(CRED);
ListObjectsRequest lor = new ListObjectsRequest()
                             .withBucketName("myBucket")
                             .withPrefix("code/");
ObjectListing objectListing = s3.listObjects(lor);
for (S3ObjectSummary summary: objectListing.getObjectSummaries()) {
    System.out.println(summary.getKey());
}

I get:

code/ 
code/03000000-0001-0000-0000-000000000000/ 
code/03000000-0001-0000-0000-000000000000/special.js 
code/03000000-0001-0000-0000-000000000000/test.js 
code/03000000-0002-0000-0000-000000000000/ 

Which is exactly what I would expect. If I add a delimiter though, so that I only list the content directly under "code/" I now don't get any sub "directories" back.

Change line above (add withDelimiter() on the end) to:

ListObjectsRequest lor = new ListObjectsRequest().withBucketName("myBucket")
                                                 .withPrefix("code/")
                                                 .withDelimiter("/");

And I now only get:

code/ 

I know that S3 doesn't have "directories", instead delimited keys, but this behaviour seems odd? How would I list what is only immediately below "code"?

Answer 1

Where you have keys that have no content S3 considers them "Common Prefixes":

http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/ObjectListing.html#getCommonPrefixes%28%29

public List getCommonPrefixes()

Gets the common prefixes included in this object listing. Common prefixes are only present if a delimiter was specified in the original request.

Each common prefix represents a set of keys in the S3 bucket that have been condensed and omitted from the object summary results. This allows applications to organize and browse their keys hierarchically, similar to how a file system organizes files into directories.

For example, consider a bucket that contains the following keys:

"foo/bar/baz"
"foo/bar/bash"
"foo/bar/bang"
"foo/boo"

If calling listObjects with the prefix="foo/" and the delimiter="/" on this bucket, the returned S3ObjectListing will contain one entry in the common prefixes list ("foo/bar/") and none of the keys beginning with that common prefix will be included in the object summaries list.

Returns: The list of common prefixes included in this object listing, which might be an empty list if no common prefixes were found.

Answer 2

You can specify any directory hierarchy that lists buckets. When prefix is ​​set to "" or "/", it indicates the first-level directory of the bucket. When setting a specific subdirectory, the list of keys of the subdirectory is displayed.

Reference to Charles Menguy's answer in 'Amazon S3 listing “directories”' , and added some modifications.

  public List<String> listKeysInBucket(String bucketName, String prefix) {
    Boolean isTopLevel = false;
    String delimiter = "/";
    if(prefix == "" || prefix == "/") {
      isTopLevel = true;
    }
    if (!prefix.endsWith(delimiter)) {
      prefix += delimiter;
    }

    ListObjectsRequest listObjectsRequest = null;
    if (isTopLevel) {
      listObjectsRequest =
          new ListObjectsRequest().withBucketName(bucketName).withDelimiter(delimiter);
    } else {
      listObjectsRequest = new ListObjectsRequest().withBucketName(bucketName).withPrefix(prefix)
          .withDelimiter(delimiter);
    }
    ObjectListing objects = s3Client.listObjects(listObjectsRequest);
    return objects.getCommonPrefixes();
  }

Answer 3

This below code worked for me to list all directories in s3.

private static String bucket_name = "";
private static String secret_key = "";
private static String access_key = "";
private static String Regions region = Regions.SELECT_REGION;

public static void main(String[] args) {
        System.out.println(listKeysInBucket(bucket_name, "/"));
    }

    public static List<String> listKeysInBucket(String bucketName, String prefix) {
        boolean isTopLevel = false;
        String delimiter = "/";
        if (prefix.equals("") || prefix.equals(delimiter)) {
            isTopLevel = true;
        }
        if (!prefix.endsWith(delimiter)) {
            prefix += delimiter;
        }

        ListObjectsRequest listObjectsRequest = null;
        if (isTopLevel) {
            listObjectsRequest = new ListObjectsRequest().withBucketName(bucketName).withDelimiter(delimiter);
        } else {
            listObjectsRequest = new ListObjectsRequest().withBucketName(bucketName).withPrefix(prefix)
                    .withDelimiter(delimiter);
        }

        ObjectListing objects = s3Client().listObjects(listObjectsRequest);
        return objects.getCommonPrefixes();
    }

    public static AmazonS3 s3Client() {
        AWSCredentials s3Configs = new BasicAWSCredentials(access_key,secret_key);
        return AmazonS3ClientBuilder.standard()
                .withCredentials(new AWSStaticCredentialsProvider(s3Configs)).withRegion(region )
                .build();
    }