Apache Beam S3 filesystem extension always requires aws region input even in other pipelines within my project that don't us AWS

Question

I have a project written in java with multiple beam pipelines within it that I compile to a jar file for execution on a server. Everything works currently where I'm just reading from GCP resources but I just added a pipeline that writes to S3. The S3 part works independently but now when I try to run the other pipelines that just use GCP, it throws an exception because I'm not providing S3 options (even though I don't need them) - the the error message below. It seems a little off that I need to specify an AWS region when I'm only using GCP resources (or maybe I'm doing something wrong). Is there a way to only register the filesystems that I'm using for a specific pipeline rather than a blanket register all filesystems on initialization?

INFO: The AWS S3 Beam extension was included in this build, but the awsRegion flag was not specified. If you don't plan to use S3, then ignore this message.

It throws this (above) info warning as if its possible to ignore it the AWS region but then throws an Exception (below).

Exception in thread "main" com.amazonaws.SdkClientException: Could not find region information for 'null' in SDK metadata.

I'm packaging my Jar file using Maven then I execute a pipeline by passing in the specific main for that pipeline (ie ). Here is the stack trace I get when I try to run my pipeline that does not use AWS or S3 at all, only GCP.

Jan 08, 2019 4:14:00 PM org.apache.beam.sdk.io.aws.s3.S3FileSystem <init>
INFO: The AWS S3 Beam extension was included in this build, but the awsRegion flag was not specified. If you don't plan to use S3, then ignore this message.
Exception in thread "main" com.amazonaws.SdkClientException: Could not find region information for 'null' in SDK metadata.
    at com.amazonaws.client.builder.AwsClientBuilder.getRegionObject(AwsClientBuilder.java:256)
    at com.amazonaws.client.builder.AwsClientBuilder.withRegion(AwsClientBuilder.java:243)
    at org.apache.beam.sdk.io.aws.s3.DefaultS3ClientBuilderFactory.createBuilder(DefaultS3ClientBuilderFactory.java:42)
    at org.apache.beam.sdk.io.aws.s3.S3FileSystem.<init>(S3FileSystem.java:112)
    at org.apache.beam.sdk.io.aws.s3.S3FileSystemRegistrar.fromOptions(S3FileSystemRegistrar.java:39)
    at org.apache.beam.sdk.io.FileSystems.verifySchemesAreUnique(FileSystems.java:489)
    at org.apache.beam.sdk.io.FileSystems.setDefaultPipelineOptions(FileSystems.java:479)
    at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:47)
    at org.apache.beam.sdk.Pipeline.create(Pipeline.java:145)
    at foo.GCSPipeline.runGCSPipeline(GCSPipeline.java:192)
    at foo.GCSPipeline.main(GCSPipeline.java:239)

Answer 1

这是错误，正在此处跟踪： https : //issues.apache.org/jira/browse/BEAM-6266