![](/img/trans.png)
[英]Read a file using Java from an S3 bucket and HTTP PUT file to presigned AWS S3 URL of another bucket in a way that simulates an actual file upload
[英]Put/read files on S3 bucket using JAVA
我正在嘗試將文件放在 S3 上並使用 JAVA 從 S3 存儲桶中讀取文件,但遇到無法執行 HTTP 請求。
當我嘗試列出桶時,它工作正常。
對我來說,看起來只有 listBuckets() 方法在工作,而所有其他方法,如 putObject()、listObjects()、createBucket() 等都拋出相同的錯誤無法執行 HTTP 請求
這是我正在修復的代碼
package test;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.Bucket;
import com.amazonaws.services.s3.model.ObjectListing;
import com.amazonaws.services.s3.model.S3ObjectSummary;
import com.amazonaws.services.s3.model.PutObjectRequest;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.SdkClientException;
public class JavaS3Conn {
//private static Logger log = Logger.getLogger(JavaS3Conn.class);
public static void main(String[] args) {
String access_key = "access_key";
String secret_key = "secret_key";
String end_point = "http://end_point:port/";
String bucketName = "bucketName";
String stringObjKeyName = "Hello_S3.txt";
String fileObjKeyName = "Hello_S3.txt";
String fileName = "C:\\Users\\Desktop\\hello_S3.txt";
try {
BasicAWSCredentials creds = new BasicAWSCredentials(access_key, secret_key);
ClientConfiguration clientConfig = new ClientConfiguration()
.withProxyHost("wdctestlab-ecs1-node1.systems.uk.hsbc").withProxyPort(9020)
.withNonProxyHosts("");
AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withClientConfiguration(clientConfig).withCredentials(new AWSStaticCredentialsProvider(creds)).withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(end_point,Regions.DEFAULT_REGION.getName())).build();
/*
if(s3Client.doesBucketExistV2(bucketName)) {
System.out.println("Bucket name is not available."
+ " Try again with a different Bucket name.");
return;
}*/
//s3Client.createBucket(bucketName);
ObjectListing objectListing = s3Client.listObjects(bucketName);
for(S3ObjectSummary os : objectListing.getObjectSummaries()) {
System.out.println(os.getKey());
}
/*
s3Client.putObject(
bucketName,
stringObjKeyName,
"Uploaded String Object"
);
*/
PutObjectRequest request = new PutObjectRequest(bucketName, fileObjKeyName, new File(fileName));
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentType("plain/text");
metadata.addUserMetadata("x-amz-meta-title", "someTitle");
request.setMetadata(metadata);
//s3Client.putObject(request);
//s3Client.listObjects(bucketName);
for (Bucket bucket : s3Client.listBuckets()) {
System.out.println(" - " + bucket.getName());
}
} catch (AmazonServiceException e) {
e.printStackTrace();
} catch (SdkClientException e) {
e.printStackTrace();
}
}
}
錯誤 -
com.amazonaws.SdkClientException: Unable to execute HTTP request: bucket.endpoint
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1175
)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1121)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4914)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4860)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4854)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:880)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:848)
at test.JavaS3Conn.main(JavaS3Conn.java:51)
Caused by: java.net.UnknownHostException: bucket.endpoint
at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)
at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:111)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
at com.amazonaws.http.conn.$Proxy3.connect(Unknown Source)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1297)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
似乎我必須添加客戶端配置來解決我無法執行 HTTP 的問題
ClientConfiguration clientConfig = new ClientConfiguration()
.withProxyHost("wdctestlab-ecs1-node1.systems.uk.hsbc").withProxyPort(9020)
.withNonProxyHosts("");
我在 Mac 上的 localstack(devlocal s3 模擬器)docker 容器上運行 s3 並收到此錯誤(“SdkClientException:無法執行 HTTP 請求:bucketname.localhost”、“UnknownHostException”)。 和 listBuckets 工作。
把它放在 /etc/hosts 中為我修復了它:
127.0.0.1 bucketname.localhost
以下替代解決方案也為我修復了它,但此路徑樣式訪問不適用於 2020 年 9 月之后創建的真實 s3 存儲桶(與 localstack s3 存儲桶相反):
.withPathStyleAccessEnabled(true)
(在 AmazonS3ClientBuilder 設置行上添加)
為什么:
注意 bucket.endpoint 上的 UnknownHostException - 為什么它試圖將存儲桶放在 DNS 名稱中?:
com.amazonaws.SdkClientException: Unable to execute HTTP request: bucket.endpoint
...
Caused by: java.net.UnknownHostException: bucket.endpoint
對我來說,它是 bucketname.localhost,因為我試圖連接到在 localhost 上運行的 localstack S3 端點。
新的 S3 存儲桶命名使用虛擬托管樣式命名,其中存儲桶位於路徑的前面。 bucketname.s3.amazonaws.com
不是s3.amazonaws.com/bucketname
虛擬托管樣式是默認設置。
AWS 已棄用路徑樣式命名。 2020 年 9 月之后創建的所有存儲桶都不能使用路徑樣式。 更多信息: https : //aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story
localstack 已經實現了對虛擬主機樣式的支持,但是如果您在 docker 容器中本地運行它,則必須為每個存儲桶名稱創建主機別名。 https://github.com/localstack/localstack/issues/2631
在 linux 中運行 docker 時(並使用 systemd-resolved - Fedora/Ubuntu 做),它可以工作。 我認為這是因為以下行適用於 linux 而不適用於 mac: ping abc.localhost
(其中 abc 可以是任何東西)。 這稱為本地主機子域。
對於 mac(或不使用 systemd-resolved 的 linux),另一種選擇是向 /etc/hosts 添加行或使用brew install dnsmasq
(dnsmasq 也在 linux 上) - https://serverfault.com/questions/118378/in-my -etc-hosts-file-on-linux-osx-how-do-i-do-a-wildcard-subdomain#118589 。
Linux systemd-resolved localhost-subdomain 信息來源: https : //unix.stackexchange.com/questions/401966/how-does-every-subdomain-of-localhost-point-to-localhost-on-fedora
我在此位置跟蹤了虛擬主機主機名:(sdk 版本 1.11)
AmazonS3Client.doesObjectExist
getObjectMetadata
createRequest
resolveRequestEndpoint
new S3RequestEndpointResolver(..)
S3RequestEndpointResolver.resolveRequestEndpoint
shouldUseVirtualAddressing
BucketNameUtils.isDNSBucketName
isValidV2BucketName
isValidV2BucketName
request.setEndpoint(convertToVirtualHostEndpoint(endpoint, bucketName));
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.