Flink 无法向本地堆栈上的 S3 存储桶发送 http 请求

Question

I'm trying to call the getObject api to get an external file on S3 bucket0 within my Flink job, but it keeps getting a SdkException from my localstack setup:

 org.apache.flink.kinesis.shaded.com.amazonaws.SdkClientException:
 Unable to execute HTTP request: mybucketName.s3.localstack

This is how I created my resources with docker compose and localstack

container_name: localstack
image: localstack/localstack:0.12.15
ports:
  - "14566:4566"
expose:
  - "4566"
environment:
  - DEFAULT_REGION=us-east-1
  - AWS_ACCESS_KEY_ID=test
  - AWS_SECRET_ACCESS_KEY=test
  - EDGE_PORT=4566
  - SERVICES=s3
  - AWS_CBOR_DISABLE=1

I have tried to replace the endpoint with http://localstack:4566 and http://s3.localstack:4566 , but I still see the error

import org.apache.flink.kinesis.shaded.com.amazonaws.services.s3.AmazonS3ClientBuilder;
    
AwsClientBuilder.EndpointConfiguration endpointConfiguration =
        new AwsClientBuilder.EndpointConfiguration(
            "http://s3.localstack:4566",   // also tried localstack:4566 and 127.0.0.1:4566 etc..
            region);

s3Client = AmazonS3ClientBuilder.standard()
        .withEndpointConfiguration(endpointConfiguration)
        .build()


S3Object s3Object = s3Client.getObject(bucketName, objectLocation);

Anyone knows why I'm getting this issue? Sorry the SdkException only shows unable to execute HTTP and does not output lots for context/info. Thank you.

Answer 1

I think I have found the solution, I should call.enablePathStyleAccess() in my client

    AmazonS3ClientBuilder.standard()
        .withEndpointConfiguration(endpointConfiguration)
        .enablePathStyleAccess()
        .build()

Please refer to this post