AWS CLI and BOTO3 SDK (permissions issue)

Question

I'm able to download a s3 folder with aws cli like this:

aws s3 cp --recursive s3://bucketname/custom-folder /tmp

But I can't list folder files from boto3 library:

import boto3

s3 = boto3.resource('s3')
bucket = s3.Bucket('my-bucket')
for obj in bucket.objects.all():
    print(obj.key)

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

However, I can download a specific file if I know the specific path, which I know because the bash command works correctly:

s3.download_file(Bucket='mybucket',Key='custom-folder/1.gz',Filename='/tmp/1.gz')

My question is:

How can I download one folder recursively in Boto3 without permissions for reading/listing files in the S3 Bucket ?????

Thanks!

Answer 1

The --recursive command actually needs to list files to know which files to copy. Therefore, it would appear that you have been granted permission to list files, but only within custom-folder .

You can test this with:

aws s3 ls s3://bucketname/custom-folder

If that works, you could modify your Python app by using:

for obj in bucket.objects.filter(Prefix='custom-folder/'):

This should limit the listing request to just that folder.

If that has a problem, you might need to use the client version of the API call rather than the resource version.