Error when running the WordCount example pipeline on Dataflow with Eclipse

Question

When trying to run the WordCount example pipeline using Dataflow under Eclipse IDE, I get the following error:

Exception in thread "main" java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
    at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
    at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
    at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
    at org.apache.beam.sdk.Pipeline.create(Pipeline.java:150)
    at com.google.cloud.dataflow.examples.WordCount.main(WordCount.java:178)

Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
    ... 4 more

Caused by: java.lang.IllegalArgumentException: Missing object or bucket in path: 'gs://mysite-ga-datastreaming-196008-my-bucket/', did you mean: 'gs://some-bucket/mysite-ga-datastreaming-196008-my-bucket'?
    at org.apache.beam.sdks.java.extensions.google.cloud.platform.core.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:383)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.verifyPath(GcsPathValidator.java:77)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.validateOutputFilePrefixSupported(GcsPathValidator.java:60)
    at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:246)
    ... 9 more

Some people suggest that the error is due to the Java version, as it seems that Beam doesn't work fine with Java 9. Anyway, I'm still using Java 8. On the other hand, some other people say that the error is caused because you have to provide a subfolder under your bucket as the storage location. I've tried, but it still does not work.

If anyone faced this same issue before or can provide any advice on the error, it would be appreciated.

Answer 1

在使用管道之前，您应该在Google Cloud Storage中创建存储桶gs://mysite-ga-datastreaming-196008-my-bucket/ 。

Answer 2

Hi Mangu' suggestion is correct. You need to assign a folder instead of bucket name only for the cloud storage staging location.

Refer to my post here about all the details: link