Put/read files on S3 bucket using JAVA

Question

I am trying to put file on S3 and read files from S3 buckets using JAVA but encounter Unable to execute HTTP request.

while i tried to list the buckets then it works fine.

to me it look like only listBuckets() method is working while all other methods like putObject(), listObjects(), createBucket() etc are throwing same error Unable to execute HTTP request

here is my code which i am working on to fix i have used many methods but none works other than listBuckets() --

package test;

import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.Bucket;
import com.amazonaws.services.s3.model.ObjectListing;
import com.amazonaws.services.s3.model.S3ObjectSummary;
import com.amazonaws.services.s3.model.PutObjectRequest;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.SdkClientException;

 public class JavaS3Conn {

//private static Logger log = Logger.getLogger(JavaS3Conn.class);

public static void main(String[] args) {

    String access_key = "access_key";
    String secret_key = "secret_key";
    String end_point = "http://end_point:port/";
    String bucketName = "bucketName";
    String stringObjKeyName = "Hello_S3.txt";
    String fileObjKeyName = "Hello_S3.txt";
    String fileName = "C:\\Users\\Desktop\\hello_S3.txt";
    try {
    BasicAWSCredentials creds = new BasicAWSCredentials(access_key, secret_key); 
ClientConfiguration clientConfig = new ClientConfiguration()
                .withProxyHost("wdctestlab-ecs1-node1.systems.uk.hsbc").withProxyPort(9020)
                .withNonProxyHosts("");
    AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withClientConfiguration(clientConfig).withCredentials(new AWSStaticCredentialsProvider(creds)).withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(end_point,Regions.DEFAULT_REGION.getName())).build();
    /*
    if(s3Client.doesBucketExistV2(bucketName)) {
        System.out.println("Bucket name is not available."
          + " Try again with a different Bucket name.");
        return;
    }*/


    //s3Client.createBucket(bucketName);

    ObjectListing objectListing = s3Client.listObjects(bucketName);
    for(S3ObjectSummary os : objectListing.getObjectSummaries()) {
        System.out.println(os.getKey());
    }
    /*
    s3Client.putObject(
              bucketName, 
              stringObjKeyName, 
              "Uploaded String Object"
            );
    */
    PutObjectRequest request = new PutObjectRequest(bucketName, fileObjKeyName, new File(fileName));
    ObjectMetadata metadata = new ObjectMetadata();
    metadata.setContentType("plain/text");
    metadata.addUserMetadata("x-amz-meta-title", "someTitle");
    request.setMetadata(metadata);
    //s3Client.putObject(request);
    //s3Client.listObjects(bucketName);

    for (Bucket bucket : s3Client.listBuckets()) {
        System.out.println(" - " + bucket.getName());
    }

    } catch (AmazonServiceException e) {
        e.printStackTrace();
    } catch (SdkClientException e) {
        e.printStackTrace();            

    }

}

}

Error --

 com.amazonaws.SdkClientException: Unable to execute HTTP request: bucket.endpoint
    at 
 com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1175 
     )
         at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1121)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4914)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4860)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4854)
    at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:880)
    at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:848)
    at test.JavaS3Conn.main(JavaS3Conn.java:51)
Caused by: java.net.UnknownHostException: bucket.endpoint
    at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
    at java.net.InetAddress.getAllByName(InetAddress.java:1192)
    at java.net.InetAddress.getAllByName(InetAddress.java:1126)
    at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)
    at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:111)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
    at com.amazonaws.http.conn.$Proxy3.connect(Unknown Source)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
    at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1297)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)

It seems like i have to add client config which resolve my the issue of unable to execute HTTP

ClientConfiguration clientConfig = new ClientConfiguration()
                .withProxyHost("wdctestlab-ecs1-node1.systems.uk.hsbc").withProxyPort(9020)
                .withNonProxyHosts("");

Answer 1

I am running s3 on a localstack (a devlocal s3 emulator) docker container on a Mac and got this error ("SdkClientException: Unable to execute HTTP request:bucketname.localhost", "UnknownHostException"). And listBuckets works.

Putting this in /etc/hosts fixed it for me:

127.0.0.1       bucketname.localhost

The following alternative solution also fixed it for me but this path-style-access will not work with real s3 buckets (as opposed to localstack s3 buckets) created after Sep 2020:

.withPathStyleAccessEnabled(true)

(added on the AmazonS3ClientBuilder setup line)

Why:

Note the UnknownHostException on bucket.endpoint - why is it trying to put the bucket in the DNS name?:

com.amazonaws.SdkClientException: Unable to execute HTTP request: bucket.endpoint
...
Caused by: java.net.UnknownHostException: bucket.endpoint

For me it was bucketname.localhost since I was trying to connect to a localstack S3 end-point running on localhost.

The new S3 bucket naming uses the virtual hosted style naming where the bucket goes on the front of the path. bucketname.s3.amazonaws.com not s3.amazonaws.com/bucketname

Virtual hosted style is the default.

AWS has deprecated path-style naming. All buckets created after September 2020 can't use path-style. More info: https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story

localstack has implemented support for virtual host style but if you are running it locally in a docker container, you have to create host aliases for each bucket name. https://github.com/localstack/localstack/issues/2631

When running docker in ​linux (and using systemd-resolved - Fedora/Ubuntu do), it works. I think this is because the following line works in linux and not on mac: ping abc.localhost (where abc can be anything). This is called localhost subdomains.

For mac (or linux not using systemd-resolved), another option is adding lines to /etc/hosts or using brew install dnsmasq (dnsmasq is also on linux) - https://serverfault.com/questions/118378/in-my-etc-hosts-file-on-linux-osx-how-do-i-do-a-wildcard-subdomain#118589 .

Linux systemd-resolved localhost-subdomain information source: https://unix.stackexchange.com/questions/401966/how-does-every-subdomain-of-localhost-point-to-localhost-on-fedora

I traced the virtual-host hostname addition to this spot: (sdk version 1.11)

AmazonS3Client.doesObjectExist
 getObjectMetadata
  createRequest
   resolveRequestEndpoint
    new S3RequestEndpointResolver(..)

S3RequestEndpointResolver.resolveRequestEndpoint
 shouldUseVirtualAddressing
  BucketNameUtils.isDNSBucketName
   isValidV2BucketName
    isValidV2BucketName
 request.setEndpoint(convertToVirtualHostEndpoint(endpoint, bucketName));