无法使用 boto3 在 Scaleway 上列出存储桶对象

Question

I saw a few similar posts, but unfortunately none helped me. I have an s3 bucket (on scaleway), and I'm trying to simply list all objects contained in that bucket, using boto3 s3 client as follow:

s3 = boto3.client('s3',
                  region_name=AWS_S3_REGION_NAME,
                  endpoint_url=AWS_S3_ENDPOINT_URL,
                  aws_access_key_id=AWS_ACCESS_KEY_ID,
                  aws_secret_access_key=AWS_SECRET_ACCESS_KEY
                  )

all_objects = s3.list_objects_v2(Bucket=AWS_STORAGE_BUCKET_NAME)

This simple piece of code responds with an error:

botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the ListObjects operation: The specified key does not exist.

First, the error seems inapropriate to me since I'm not specifying any key to search. I also tried to pass a Prefix argument to this method to narrow down the search to a specific subdirectory, same error. Second, I tried to achieve the same thing using boto3 Resource rather than Client , as follow:

session = boto3.Session(
    region_name=AWS_S3_REGION_NAME,
    aws_access_key_id=AWS_ACCESS_KEY_ID,
    aws_secret_access_key=AWS_SECRET_ACCESS_KEY
)
resource = session.resource(
    's3',
    endpoint_url=AWS_S3_ENDPOINT_URL,
)

for bucket in resource.buckets.all():
    print(bucket.name)

That code produces absolutely nothing. One weird thing that strikes me is that I don't pass the bucket_name anywhere here, which seems to be normal according to aws documentation

There's no chance that I misconfigured the client, since I'm able to use the put_object method perfectly with that same client. One strange though: when I want to put a file, I pass the whole path to put_object as Key (as I found it to be the way to go), but the object is inserted with the bucket name prepend to it. So let's say I call put_object(Key='/path/to/myfile.ext') , the object will end up to be /bucket-name/path/to/myfile.ext .

Is this strange behavior the key to my problem ? How can I investigate what's happening, or is there another way I could try to list bucket files ?

Thank you

EDIT: So, after logging the request that boto3 client is sending, I noticed that the bucket name is append to the url, so instead of requesting https://<bucket_name>.s3.<region>.<provider>/ , it requests https://<bucket_name>.s3.<region>.<provider>/<bucket-name>/ , which is leading to the NoSuchKey error. I took a look into the botocore library, and I found this:

url = _urljoin(endpoint_url, r['url_path'], host_prefix)

in botocore.awsrequest line 252, where r['url_path'] contains /skichic-bucket?list-type=2 . So from here, I should be able to easily patch the library core to make it work for me.

Plus, the Prefix argument is not working, whatever I pass into it I always receive the whole bucket content, but I guess I can easily patch this too.

Now it's not satisfying, since there's no issue related to this on github, I can't believe that the library contains such a bug that I'm the first one to encounter.

Does anyone can explain this whole mess ? >.<

Answer 1

you can try this. you'll have to use your resource instead of my s3sr.

s3sr = resource('s3')
bucket = 'your-bucket'
prefix = 'your-prefix/' # if no prefix, pass ''

def get_keys_from_prefix(bucket, prefix):
    '''gets list of keys for given bucket and prefix'''
    keys_list = []
    paginator = s3sr.meta.client.get_paginator('list_objects_v2')
    # use Delimiter to limit search to that level of hierarchy
    for page in paginator.paginate(Bucket=bucket, Prefix=prefix, Delimiter='/'):
        keys = [content['Key'] for content in page.get('Contents')]
        print('keys in page: ', len(keys))
        keys_list.extend(keys)
    return keys_list

keys_list = get_keys_from_prefix(bucket, prefix)

Answer 2

After looking more closely into things, I've found out that (a lot) of botocore services endpoints patterns starts with the bucket name. For example, here's the definition of the list_objects_v2 service:

"ListObjectsV2":{
      "name":"ListObjectsV2",
      "http":{
        "method":"GET",
        "requestUri":"/{Bucket}?list-type=2"
      },

My guess is that in the standard implementation of AWS S3, there's a generic endpoint_url (which explains @jordanm comment) and the targeted bucket is reached through the endpoint.

Now, in the case of Scaleway, there's an endpoint_url for each bucket, with the bucket name contained in that url (eg https://<bucket_name>.s3.<region>.<provider> ), and any endpoint should directly starts with a bucket Key .

I made a fork of botocore where I rewrote every endpoint to remove the bucket name, if that can help someone in the future.

Thank's again to all contributors !

Answer 3

For those who are facing the same issue, try changing your endpoint_url parameter in your boto3 client or resource instantiation from https://<bucket_name>.s3.<region>.<provider> to https://s3.<region>.<provider> ; ie for Scaleway : https://s3.<region>.scw.cloud .

You can then set the Bucket parameter to select the bucket you want.

list_objects_v2(Bucket=<bucket_name>)