使用Apache Camel测试Localstack AWS

Question

I'm trying to leverage localstack to do local AWS mock testing. Specifically, I'd like to download a file from a localstack S3 bucket by using an Apache Camel S3 route. However, I am running into errors. Here are the steps I've taken...

I've created a test localstack S3 bucket by entering...

aws --endpoint-url=http://localhost:4572 s3 mb s3://mytestbucket

Then, I uploaded a test file to this bucket...

aws --endpoint-url=http://localhost:4572 s3 cp docker-compose.yml s3://mytestbucket

Both operations completed successfully. In my Java code, I've created a CDI Producer to produce a com.amazonaws.services.s3.AmazonS3 client object...

AWSCredentials credentials = new BasicAWSCredentials("XXXXX", "XXXXX");
AWSCredentialsProvider credentialsProvider = new AWSStaticCredentialsProvider(credentials);
AmazonS3ClientBuilder clientBuilder = AmazonS3ClientBuilder.standard()
                .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("http://localhost:4572", "us-east-1"))
                .withCredentials(credentialsProvider);

AmazonS3 s3Client = clientBuilder.build();  

Finally, I created a Camel route designed to download the contents of mytestbucket into a local directory...

<route id="s3test">
        <from
            uri="aws-s3://arn:aws:s3:::mytestbucket?amazonS3Client=#s3Client&amp;deleteAfterRead=false&amp;maxMessagesPerPoll=25&amp;delay=5000" />
        <log message="S3 consumer fired" loggingLevel="DEBUG" />
        <log message="Sending S3 file to ${in.header.CamelAwsS3Key}..." />
        <to
            uri="file://tmp/camel?fileName=${in.header.CamelAwsS3Key}" />
    </route>

However, when my application starts up and attempts to run the route, I get the following Exception:

Caused by: org.jboss.weld.exceptions.DeploymentException: Failed to create route s3test: Route(s3test)[[From[aws-s3://arn:aws:s3:::mytestbucket?amazo... because of Unable to execute HTTP request: mytestbucket.localhost at org.jboss.weld.bootstrap.events.AbstractDeploymentContainerEvent.fire(AbstractDeploymentContainerEvent.java:38) at org.jboss.weld.bootstrap.events.AfterDeploymentValidationImpl.fire(AfterDeploymentValidationImpl.java:28) at org.jboss.weld.bootstrap.WeldStartup.validateBeans(WeldStartup.java:499) at org.jboss.weld.bootstrap.WeldBootstrap.validateBeans(WeldBootstrap.java:93) at org.jboss.as.weld.WeldStartService.start(WeldStartService.java:97) at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1736) at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1698) ... 6 more Caused by: org.apache.camel.FailedToCreateRouteException: Failed to create route s3test: Route(s3test)[[From[aws-s3://ar n:aws:s3:::mytestbucket?amazo... because of Unable to execute HTTP request: mytestbucket.localhost at org.apache.camel.impl.RouteService.warmUp(RouteService.java:147) at org.apache.camel.impl.DefaultCamelContext.doWarmUpRoutes(DefaultCamelContext.java:3949) at org.apache.camel.impl.DefaultCamelContext.safelyStartRouteServices(DefaultCamelContext.java:3856) at org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRoutes(DefaultCamelContext.java:3642) at org.apache.camel.impl.DefaultCamelContext.doStartCamel(DefaultCamelContext.java:3494) at org.apache.camel.impl.DefaultCamelContext.access$000(DefaultCamelContext.java:209) at org.apache.camel.impl.DefaultCamelContext$2.call(DefaultCamelContext.java:3253) at org.apache.camel.impl.DefaultCamelContext$2.call(DefaultCamelContext.java:3249) at org.apache.camel.impl.DefaultCamelContext.doWithDefinedClassLoader(DefaultCamelContext.java:3272) at org.apache.camel.impl.DefaultCamelContext.doStart(DefaultCamelContext.java:3249) at org.apache.cam el.support.ServiceSupport.start(ServiceSupport.java:61) at org.apache.camel.impl.DefaultCamelContext.start(DefaultCamelContext.java:3165) at org.apache.camel.impl.DefaultCamelContext$Proxy$_$$_WeldClientProxy.start(Unknown Source) at org.apache.camel.cdi.CdiCamelExtension.afterDeploymentValidation(CdiCamelExtension.java:425) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:95) at org.jboss.weld.injection.MethodInvocationStrategy$SpecialParamPlusBeanManagerStrategy.invoke(MethodInvocationStrategy.java:144) at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:330) at org.jboss.weld.event.ExtensionObserverMethodImpl.sendEvent(ExtensionObserv erMethodImpl.java:123) at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:308) at org.jboss.weld.event.ObserverMethodImpl.notify(ObserverMethodImpl.java:286) at org.jboss.weld.util.Observers.notify(Observers.java:172) at org.jboss.weld.event.ObserverNotifier.notifySyncObservers(ObserverNotifier.java:285) at org.jboss.weld.event.ObserverNotifier.notify(ObserverNotifier.java:273) at org.jboss.weld.event.ObserverNotifier.fireEvent(ObserverNotifier.java:177) at org.jboss.weld.event.ObserverNotifier.fireEvent(ObserverNotifier.java:171) at org.jboss.weld.bootstrap.events.AbstractContainerEvent.fire(AbstractContainerEvent.java:53) at org.jboss.weld.bootstrap.events.AbstractDeploymentContainerEvent.fire(AbstractDeploymentContainerEvent.java:35) ... 12 more Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: mytestbucket.localhost at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1114) at com. amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1064) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4330) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4277) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4271) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:835) at org.apache.camel.component.aws.s3.S3Endpoint. doStart(S3Endpoint.java:112) at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61) at org.apache.camel.util.ServiceHelper.startService(ServiceHelper.java:75) at org.apache.camel.impl.RouteService.doWarmUp(RouteService.java:157) at org.apache.camel.impl.RouteService.warmUp(RouteService.java:145) ... 42 more Caused by: java.net.UnknownHostException: mytestbucket.localhost at java.net.InetAddress.getAllByName0(InetAddress.java:1280) at java.net.InetAddress.getAllByName(InetAddress.java:1192) at java.net.InetAddress.getAllByName(InetAddress.java:1126) at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27) at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:111) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) at sun.reflect.NativeMet hodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) at com.amazonaws.http.conn.$Proxy246.connect(Unknown Source) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) at com.amazonaws.http.apache.client.impl.SdkHttpCli ent.execute(SdkHttpClient.java:72) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1236) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056) ... 57 more

I thought that since my AmazonS3 client specifically indicates that http://localhost:4572 is my endpoint then it should not reach out to the actual S3 service. That seems to be the case, but for some reason it is trying to access mytestbucket.localhost as if it were a network host. Any idea what I could be doing wrong here or why this route/configuration isn't working?

Answer 1

For anyone else running into this problem, the answer lies in creating the client like this:

AWSCredentials credentials = new BasicAWSCredentials("XXXXX", "YYYYY");
AWSCredentialsProvider credentialsProvider = new AWSStaticCredentialsProvider(credentials);
AmazonS3ClientBuilder clientBuilder = AmazonS3ClientBuilder.standard().withPathStyleAccessEnabled(true)
                    .disableChunkedEncoding()
                    .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(S3_ENDPOINT, "us-east-1"))
                    .withCredentials(credentialsProvider);

The key is .withPathStyledAccessEnabled(true) . Adding that will generate the endpoint URL properly.