Example with Apache Beam Java SDK and PubSub Source

Question

I'm struggling to get Google Cloud Dataflow Java SDK to work with a PubSub Source. The minimalistic code below always comes back with a 401 (Unauthorized) error. I've tried to figure out how to provide the credentials but couldn't find a way to get it to work. I've used the gcloud auth login beforehand to make sure I'm logged in and the subscription exists.

The following error is thrown:

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:java (default-cli) on project my-dataflow: An exception occured while executing the Java class. null: InvocationTargetException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 401 Unauthorized
[ERROR] {
[ERROR] "code" : 401,
[ERROR] "errors" : [ {
[ERROR] "domain" : "global",
[ERROR] "message" : "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
[ERROR] "reason" : "unauthorized"
[ERROR] } ],
[ERROR] "message" : "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
[ERROR] "status" : "UNAUTHENTICATED"
[ERROR] }

My code is as follows:

public class MyDataflow{

  public static void main(String[] args) {

    PipelineOptions options = PipelineOptionsFactory.create();

    PubsubOptions dataflowOptions = options.as(PubsubOptions.class);
    dataflowOptions.setStreaming(true);

    Pipeline p = Pipeline.create(options);

    p.apply(PubsubIO.<String>read().subscription("projects/my-project/subscription/my-subscription")
            .withCoder(StringUtf8Coder.of()));

    // Run the pipeline.
    p.run().waitUntilFinish();
  }
}

Answer 1

I had to set the GOOGLE_APPLICATION_CREDENTIALS to point to my json credential file.

https://developers.google.com/identity/protocols/application-default-credentials

Answer 2

If you are running pipeline in your local then export credential. Run following command in terminal

    export GOOGLE_APPLICATION_CREDENTIALS=/CREDENTIAL_JSON_PATH/credentialFile.json

If you are debugging pipeline in intellj then either pass credential path to bash profile or

     Maven -> Go to Runner -> Environment Variable -> Add (GOOGLE_APPLICATION_CREDENTIALS=/CREDENTIAL_JSON_PATH/credentialFile.json)

and save it