[英]Transfer files from one S3 bucket to another S3 bucket using python boto3
[英]finding s3 bucket's level 1 prefix sizes while including versions using boto3 and python
我是一個 aws python 新手,並試圖考慮通過 UI 上的指標選項卡顯示的總存儲桶大小與在給定存儲桶中一次計算一個文件夾的大小。 我試圖通過設置庫存配置來獲取它,但它沒有顯示我在尋找什么。
我有一個啟用了版本控制的 s3 存儲桶名稱my_bucket 。
它有 100 個對象和 26 個子文件夾(每個子文件夾中有 100000 多個對象,每個對象至少有兩個版本)
我要做什么:計算並顯示總大小,包括 180 個子文件夾中的每個子文件夾的版本。
A Size 1GB
B Size 10TB
.
.
.
Z Size 13TB
我要怎么做 找到一個結合了
來自鏈接一的基於配置文件的身份驗證並使用bucket.object_versions
從鏈接 2 計算一級文件夾大小
同時也考慮到版本。 (Link2 沒有版本)
Link1 https://stackoverflow.com/a/58125684/4590025
Link2 https://stackoverflow.com/a/49763268/4590025
import boto3
PROFILE = "my_profile"
BUCKET = "my_bucket"
session = boto3.Session(profile_name = PROFILE)
s3 = session.resource('s3')
bucket = s3.Bucket(BUCKET)
#bucket.object_versions.do_something_with_it
conn = boto3.client('s3')
top_level_folders = dict()
for key in conn.list_objects(Bucket='my_bucket')['Contents']:
folder = key['Key'].split('/')[0]
print("Key %s in folder %s. %d bytes" % (key['Key'], folder, key['Size']))
if folder in top_level_folders:
top_level_folders[folder] += key['Size']
else:
top_level_folders[folder] = key['Size']
for folder, size in top_level_folders.items():
print("Folder: %s, size: %d" % (folder, size))
我還提到了https://stackoverflow.com/a/48867829並且我不確定如何使用 go 關於使用這兩個,目前當我運行它時,盡管設置了 Z21D6F454240CF525A89A6E ,但我得到了以下錯誤:
Traceback (most recent call last):
File ".\folder_size.py", line 17, in <module>
for key in conn.list_objects(Bucket='my_bucket')['Contents']:
File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\client.py", line 622, in _make_api_call
operation_model, request_dict, request_context)
File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\client.py", line 641, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\endpoint.py", line 132, in _send_request
request = self.create_request(request_dict, operation_model)
File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\endpoint.py", line 116, in create_request
operation_name=operation_model.name)
File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\hooks.py", line 211, in _emit
response = handler(**kwargs)
File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\signers.py", line 90, in handler
return self.sign(operation_name, request)
File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\signers.py", line 160, in sign
auth.add_auth(request)
File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\auth.py", line 357, in add_auth
raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials
PS C:\Users\ginger\test>
問題是該程序使用:
conn = boto3.client('s3')
這忽略了之前設置的配置文件:
session = boto3.Session(profile_name = PROFILE)
因此,如果您想使用配置文件創建 S3 客戶端,那么它應該使用:
conn = session.client('s3')
為避免分頁問題,您可以使用資源方法檢索所有對象:
for object in bucket.objects.all():
folder = object.key.split('/')[0]
print("Key %s in folder %s. %d bytes" % (object.key, folder, object.size))
...
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.