how to upload sub-folder which is empty to S3 using python

Question

The following code works fine except if there is a subfolder, which does not have any file inside, then the subfolder will not appear in S3. eg if /home/temp/subfolder has no file, then subfolder will not show in S3. how to change the code so that the empty folder is also uploaded in S3? I tried to write sth. (see note below), but do not know how to call put_object() to the empty subfolder.

#!/usr/bin/env python
import os
from boto3.session import Session

path = "/home/temp"
session = Session(aws_access_key_id='XXX', aws_secret_access_key='XXX')
s3 = session.resource('s3')

for subdir, dirs, files in os.walk(path):
    # note: if not files ......
    for file in files:
        full_path = os.path.join(subdir, file)
        with open(full_path, 'rb') as data:

s3.Bucket('my_bucket').put_object(Key=full_path[len(path)+1:],    
Body=data)

besides, I tried to call this function to check if a subfolder or file exist or not. it works for file, but not subfolder. how to check if a subfolder exists or not? (if a subfolder exists, I will not upload)

def check_exist(s3, bucket, key):
    try:
        s3.Object(bucket, key).load()
    except botocore.exceptions.ClientError as e:
        return False
    return True

BTW, I refer the above code from

check if a key exists in a bucket in s3 using boto3

and

http://www.developerfiles.com/upload-files-to-s3-with-python-keeping-the-original-folder-structure/

thanks them for sharing the code.

Answer 1

Directories (folders, subfolders, etc.) do not exist in S3.

When you copy this file to an empty S3 bucket /mydir/myfile.txt , only the file myfile.txt is copied to S3. The directory mydir is not created as that string is part of the file name mydir/myfile.txt . The actual file name is the full path, no subdirectories exist or are created.

S3 simulates directories by using a prefix when listing files in the bucket. If you specify mydir/ , then all of the S3 objects that start with mydir/ will be returned including objects such as mydir/anotherfolder/myotherfile.txt . S3 supports a delimitor such as / so that the appearance of subdirectories can be created.

Note: There is no / at the beginning of a file name for S3 objects.

Listing Keys Hierarchically Using a Prefix and Delimiter