检查路径是否是文件 Amazon S3

Question

I have functionality for listing AWS S3 directories with Scala and I would like to check if the listed path is a file or a directory. How can I implement this functionality (isFile method) using amazon-sdk-s3?

Here is how it looks like:

  def listContents(): Seq[T] =
    val paths = s3Client.list(inputPath)
    for {
      path <- paths if isFile(new Path(path)).getOrElse(false)
      res <- transform(path.toString)
    } yield res

def isFile(path: String) = ??? //implementation I need

Answer 1

Amazon S3 does not have the concept of a 'Directory'.

Instead, the full path of an object is stored in its Key (filename).

For example, an object can be stored in Amazon S3 with a Key of: invoices/2020-09/inv22.txt

This object can be created even if the invoices and 2020-09 directories do not exist. When viewed through the Amazon S3 console, it will appear as though those directories were automatically created, but if the object is deleted, those directories will disappear (because they never existed).

If a user clicks the "Create Folder" button in the Amazon S3 management console, a zero-length object is created with the same name as the folder . This 'forces' the folder to appear even if there are no objects 'inside' the folder. However, it is not actually a folder.

Therefore, it is not possible to "check if the listed path is a file or a directory" because directories do not exist. Instead, I recommend that you assume everything is a 'file' unless it is zero-length .

Answer 2

S3 doesn't have the notion of folders commonly found in file systems but instead has a flat structure, more details can be found here .

Generally speaking elements that don't end in "/" are to be treated as objects but, while the AWS Web console doesn't allow you to upload files that end in "/", this is possible via SDK/API:

The Amazon S3 console treats all objects that have a forward slash ("/") character as the last (trailing) character in the key name as a folder, for example examplekeyname/ . You can't upload an object that has a key name with a trailing "/" character using the Amazon S3 console. However, you can upload objects that are named with a trailing "/" with the Amazon S3 API by using the AWS CLI, AWS SDKs, or REST API.

The other answer suggests to assume everything is a file unless it's zero length, this is a good suggestion but will break apart in case some of your files are expected to be empty (but you still need to process them), below an example of the response metadata returned by an empty txt file I just uploaded and tried to retrieve:

{
  "HTTPHeaders": {
    "accept-ranges": "bytes",
    "content-length": "0",
    "content-type": "text/plain",
    "date": "Mon, 14 Sep 2020 10:14:29 GMT",
    "etag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
    "last-modified": "Mon, 14 Sep 2020 10:13:52 GMT",
    "x-amz-server-side-encryption": "AES256"
  },
  "HTTPStatusCode": 200,
  "RetryAttempts": 0
}

Depending on how you traverse/list your objects, chances are that the response already includes objects only, but in case there's any ambiguity I'd suggest you to still attempt to retrieve the object and be prepared to handle an exception. If you try to retrive a key that is a folder, Amazon S3 will return an HTTP status code 404 ("no such key") error - Docs here .

Answer 3

It is possible to check if a folder does exist. What I have done is using listObjectV2 for it.

A bit of tricks here

1. Appending '/' to the folder name and then pass it to listObjectV2.
2. Setting max item return is 1

When you got the list object backs you can easily check if the object (folder) exists or not.

I will update my code a bit when I have time.