Boto3 folder sync under new S3 'folder'

Question

So, before anyone tells me about the flat structure of S3, I already know, but the fact is you can create 'folders' in S3. My objective with this Python code is to create a new folder named using the date of running and appending the user's input to this (which is the createS3Folder function) - I then want to sync a folder in a local directory to this folder.

The problem is that my upload_files function creates a new folder in S3 that exactly emulates the folder structure of my local set up.

Can anyone suggest how I would just sync the folder into the newly created one without changing names?

import sys
import boto3
import datetime
import os

teamName = raw_input("Please enter the name of your project: ")
bucketFolderName = ""

def createS3Folder():
    date = datetime.date.today().strftime("%Y") + "." + 
    datetime.date.today().strftime("%B") + "." + 
    datetime.date.today().strftime("%d")
    date1 = datetime.date.today()
    date = str(date1) + "/" #In order to generate a file, you must 
    put "/" at the end of key
    bucketFolderName = date + teamName + "/"  
    client = boto3.client('s3')
    client.put_object(Bucket='MY_BUCKET',Key=bucketFolderName)  
    upload_files('/Users/local/directory/to/sync')

def upload_files(path):
    session = boto3.Session()
    s3 = session.resource('s3')
    bucket = s3.Bucket('MY_BUCKET')
    for subdir, dirs, files in os.walk(path):
        for file in files:
            full_path = os.path.join(subdir, file)
            with open(full_path, 'rb') as data:
                bucket.put_object(Key=bucketFolderName, Body=data)

def main():
    createS3Folder()

if __name__ == "__main__":
    main()

Answer 1

Your upload_files() function is uploading to:

bucket.put_object(Key=bucketFolderName, Body=data)

This means that the filename ("Key") on S3 will be the name of the 'folder'. It should be:

 bucket.put_object(Key=bucketFolderName + '/' + file, Body=data)

The Key is the full path of the destination object, including the filename (not just a 'directory').

In fact, there is no need to create the 'folder' beforehand -- just upload to the desired Key.

If you are feeling lazy, use the AWS Command-Line Interface (CLI) aws s3 sync command to do it for you!

Answer 2

"the fact is you can create 'folders' in S3"

No, you can't.

You can create an empty object that looks like a folder in the console, but it is still not a folder, it still has no meaning, it is still unnecessary, and if you delete it via the API, all the files you thought were "in" the folder will still be in the bucket. (If you delete it from the console, all the contents are deleted from the bucket, because the console explicitly deletes every object starting with that key prefix.)

The folder you are creating is not a container and cannot have anything inside it, because S3 does not have folders that are containers.

If I want to store a file cat.png and make it look like it's in the hat/ folder, you simply set the object key to hat/cat.png . This has exactly the same effect as observed in the console, whether or not the hat/ folder was explicitly created or not.

To so what you want, you simply build the desired object key for each object with string manipulation, including your common prefix ("folder name") and / delimiters. Any folder structure the / delimiters imply will be displayed in the console as a result.