[英]parallell copy of buckets/keys from boto3 or boto api between 2 different accounts/connections
[英]How to access keys from buckets with periods (.) in their names using boto3?
我正在尝试获取所有存储桶的加密状态以获取安全报告。 但是,由于加密是基于密钥级别的,因此我想迭代所有密钥并获得通用加密状态。 例如,“是”是所有密钥都被加密,如果没有加密则为“否”,而“部分”是加密的。
我必须使用boto3,因为boto存在已知问题,其中每个密钥的加密状态始终返回None。 看这里。
我试图使用boto3迭代我的每个桶中的所有键。 以下代码可以正常工作,直到它遇到包含句点的名称的存储桶,例如“my.test.bucket”。
from boto3.session import Session
session = Session(aws_access_key_id=<ACCESS_KEY>,
aws_secret_access_key=<SECRET_KEY>,
aws_session_token=<TOKEN>)
s3_resource = session.resource('s3')
for bucket in s3_resource.buckets.all():
for obj in bucket.objects.all():
key = s3_resource.Object(bucket.name, obj.key)
# Do some stuff with the key...
当它遇到名称中有句点的存储桶时,它会在bucket.objects.all()
时抛出此异常,告诉我将所有请求发送到特定端点。 可以在抛出的异常对象中找到此端点。
for obj in bucket.objects.all():
File "/usr/local/lib/python2.7/site-packages/boto3/resources/collection.py", line 82, in __iter__
for page in self.pages():
File "/usr/local/lib/python2.7/site-packages/boto3/resources/collection.py", line 165, in pages
for page in pages:
File "/usr/lib/python2.7/dist-packages/botocore/paginate.py", line 85, in __iter__
response = self._make_request(current_kwargs)
File "/usr/lib/python2.7/dist-packages/botocore/paginate.py", line 157, in _make_request
return self._method(**current_kwargs)
File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 310, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 395, in _make_api_call
raise ClientError(parsed_response, operation_name)botocore.exceptions.ClientError: An error occurred (PermanentRedirect) when calling the ListObjects operation: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
s3_resource = session.resource('s3', endpoint_url='my.test.bucket.s3.amazonaws.com')
s3_resource = session.resource('s3', region_name='eu-west-1')
我相信问题类似于boto中的这个stackoverflow问题 ,它通过在s3Connection构造函数中设置calling_format参数来解决问题。 不幸的是,我不能使用boto(见上文)。
这是最终为我工作的东西。 它不是最优雅的方法,但它起作用=)。
from boto3.session import Session
session = Session(aws_access_key_id=<ACCESS_KEY>,
aws_secret_access_key=<SECRET_KEY>,
aws_session_token=<TOKEN>)
s3_resource = session.resource('s3')
# First get all the bucket names
bucket_names = [bucket.name for bucket in s3_resource.buckets.all()]
for bucket_name in bucket_names:
# Check each name for a "." and use a different resource if needed
if "." in bucket_name:
region = session.client('s3').get_bucket_location(Bucket=bucket_name)['LocationConstraint']
resource = session.resource('s3', region_name=region)
else:
resource = s3_resource
bucket = resource.Bucket(bucket_name)
# Continue as usual using this resource
for obj in bucket.objects.all():
key = resource.Object(bucket.name, obj.key)
# Do some stuff with the key...
只是概括了Ben提供的好答案。
import boto3
knownBucket = 'some.topLevel.BucketPath.withPeriods'
s3 = boto3.resource('s3')
#get region
region = s3.meta.client.get_bucket_location(Bucket=knownBucket)['LocationConstraint']
#set region in resource
s3 = boto3.resource('s3',region_name=region)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.