Cannot Access Subfolder of S3 bucket – Python, Boto3

Question

I have been given access to a subfolder of an S3 bucket, and want to access all files inside using Python and boto3. I am new to S3 and have read the docs to death, but haven't been able to figure out how to successfully access just one subfolder. I understand that s3 does not use unix-like directory structure, but I don't have access to the root bucket.

How can I configure boto3 to just connect to this subfolder?

I have successfully used this AWS CLI command to download the entire subfolder to my machine:

aws s3 cp --recursive s3://s3-bucket-name/SUB_FOLDER/ /Local/Path/Where/Files/Download/To --profile my-profile

This code:

AWS_BUCKET='s3-bucket-name'
s3 = boto3.client("s3", region_name='us-east-1', aws_access_key_id=AWS_KEY_ID, aws_secret_access_key=AWS_SECRET)
response = s3.list_objects(Bucket=AWS_BUCKET)

Returns this error:

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

I have also tried specifying the 'prefix' option in the call to list_objects, but this produces the same error.

Answer 1

You want to aws configure and save have your credentials and region then using boto3 is simple and easy.

Use boto3.resource and get the client like this:

s3_resource = boto3.resource('s3')
s3_client   = s3_resource.meta.client
s3_client.list_objects(Bucket=AWS_BUCKET)

You should be good to go.