列出来自 Amazon S3 存储桶的对象

Question

I'm planning to store a set of user related files in S3 and thinking of organizing files with prefixes similar to below in a single user-data bucket.

unique-user-guid/images/image1.jpg
unique-user-guid/images/image1.jpg
...
unique-user-guid/docs/file1.pdf
unique-user-guid/docs/file2.txt
...

My hope is to use a code similar to below to fetch all objects related to a specific user guid.

    ListObjectsV2Request request = new ListObjectsV2Request();
    request.setBucketName("user-bucket");
    request.setPrefix("unique-user-guid/");
    ListObjectsV2Result result = s3Client.listObjectsV2(request);

Is it an efficient way of organizing user specific data and list objects on demand? I'm expecting about 500k unique users in the system.

Answer 1

Your approach looks perfectly feasible .

There is a limit of 1000 objects returned per ListObjects() call, so you might need to paginate to retrieve more results. But, your quantities suggest that this will be rare.

The use of setPrefix will limit the result set quite nicely and should be quite efficient due to the way that Amazon S3 stores its index.

An alternative approach would be to store objects under a unique ID (GUID), and then use a database to keep track of the objects, names, metadata, owner, etc.