![](/img/trans.png)
[英]How can we download multiple files without folder from S3 bucket using Java SDK
[英]How to download multiple files having same prefix as filename from various folders of S3 bucket?
假設我有一個名為bucketSample
的S3存儲桶。
而且我有不同的文件夾,如abc
, def
和xyz
。
現在,在上述所有文件夾中,我都有多個帶有前綴hij_
文件。
我想下載所有帶有前綴hij_
的文件。 (例如, hij_qwe.txt
, hij_rty.pdf
等)
我經歷了各種方式,但是對於GetObject
我必須提供特定的對象名稱,而我只知道前綴。
並使用TransferManager可以下載文件夾abc
所有文件,但不能下載僅具有特定前綴的文件。
那么,有什么辦法只能下載所有帶有前綴hij_
的文件嗎?
public void getFiles(final Set<String> bucketName, final Set<String> keys, final Set<String> prefixes) {
try {
ObjectListing objectListing = s3Client.listObjects(bucketName); //lists all the objects in the bucket
while (true) {
for (Iterator<?> iterator = objectListing.getObjectSummaries().iterator();
iterator.hasNext(); ) {
S3ObjectSummary summary = (S3ObjectSummary) iterator.next();
for (String key : keys) {
for (String prefix : prefixes)
if (summary.getKey().startsWith(key + "/" prefix)) {
//HERE YOU CAN GET THE FULL KEY NAME AND HENCE DOWNLOAD IT IN NEW FILE USING THE TRANFER MANAGER
}
}
}
}
if (objectListing.isTruncated()) {
objectListing = s3Client.listNextBatchOfObjects(objectListing);
} else {
break;
}
}
} catch (AmazonServiceException e) { }
}
在此處閱讀有關AWS Directory結構的信息: AWS S3如何存儲文件? (目錄結構)
因此,對於您的用例,鍵+“ /” +前綴充當存儲在S3存儲桶中的對象的前綴。 通過比較前綴將S3存儲桶中的所有對象,您可以獲得完整的密鑰名稱。
使用python,您可以使用boto3庫,我發現它對於解決類似情況非常有用。
樣例代碼:
import boto3
import os
KEY = ''
SECRET = ''
download_folder = os.path.join(os.path.expanduser('~'), 'Downloads')
bucket = 'bucketSample'
folders = ['abc', 'def', 'xyz']
prefixes = ['hij_']
try:
# Needed for the pagination method in order to get objects with certain prefixes instead of iterating over all objects, you should get the aws_access_key_id and aws_secret_access_key for your bucket if available
s3 = boto3.resource(
's3',
aws_access_key_id=KEY,
aws_secret_access_key=SECRET)
# Needed for the download method, you should get the aws_access_key_id and aws_secret_access_key for your bucket if available
client = boto3.client(
's3',
aws_access_key_id=KEY,
aws_secret_access_key=SECRET)
# Get paginated objects
paginator = client.get_paginator('list_objects')
for folder in folders:
for file_prefix in prefixes:
prefix = folder + file_prefix
page_iterator = paginator.paginate(Bucket=bucket, Prefix=prefix)
if page_iterator:
for page in page_iterator:
if 'Contents' in page:
for content in page['Contents']:
file_path = os.path.join(download_folder, content['Key'])
s3.meta.client.download_file(bucket, str(content['Key']), file_path)
except:
print('An error occurred')
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.