如何使用java中的Apache Beam直达写入BigTable？

Question

I have been trying to get Apache Beam direct runner to write to BigTable but it seems like there is a problem.

There is no failure or confirmation errors on the terminal when I run gradle run .

My pipeline is as follows:

Pub/Sub stream of messages -> direct-runner -> BigTable

Currently using org.apache.beam.sdk.io.gcp.bigtable.BigtableIO adapter which is not working or I am doing something wrong.

There is also this another I/O adapter com.google.cloud.bigtable.beam.CloudBigtableIO and I am not sure which one to choose.

Some questions:

1. Which adapater should I use? -- this is answered, see edit.
2. How do I verify what is going wrong in writing to BigTable? Finding it hard to look into pipeline without those System.out.println statements.
3. How is authentication done to write to BigTable by direct-runner? Does the SDK automatically detect the $GOOGLE_APPLICATION_CREDENTIALS env variable & use those credentials?

Will be happy to give more details.

EDIT:

• Apache Beam Java SDK version: 2.37
• Per the docs , I should be using this connector

Answer 1

To verify what is going on:

1. add in main() BasicConfigurator.configure();

2. add in pom.xml

<dependency>
     <groupId>org.slf4j</groupId>
     <artifactId>slf4j-log4j12</artifactId>
     <version>1.7.32</version>
</dependency>

3. add this log4j.properities

# Set root logger level to DEBUG and its only appender to A1.
log4j.rootLogger=DEBUG, A1

# A1 is set to be a ConsoleAppender.
log4j.appender.A1=org.apache.log4j.ConsoleAppender

# A1 uses PatternLayout.
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n

To write on bigtable with direct runner 1. in pom.xml :

<dependency>
      <groupId>org.apache.beam</groupId>
      <artifactId>beam-runners-direct-java</artifactId>
      <version>${beam.version}</version>
      <scope>runtime</scope>
</dependency>

2. the interface for the pipeline option, to configure with run config commands:

public interface RequestsOptions extends PipelineOptions {
    @Description("File path")
    @Validation.Required
    String getInput();

    void setInput(String value);

    @Description("Output")
    @Validation.Required
    String getOutput();

    void setOutput(String value);
}

3. in run config commands:

--project=PROJECT_ID
--dataset=DATASET_NAME
--inputFile=INPUT_FILE_NAME
--region=REGION_TO_RUN //if dataflow runner
--runner=YOUR_SELECTED_RUNNER
--tempLocation=GOOGLE_STORAGE_LOCATION(to save temp files)