Bigquery to PubSub

Question

i need to push messages from bigquery to pubsbub

bigquery table contains a column with json value as a string

read that table and publish that to json

p.apply("ReadSourceBQ", BigQueryIO.readTableRows().fromQuery( "SELECT requestPayload FROM eventreplay").usingStandardSql().withTemplateCompatibility() ).apply("JSON Transform", AsJsons.of(TableRow.class)).apply("WriteToPubSub", PubsubIO.writeStrings().to("topicname"));

its not publish msg to topic.. no error too. how to convert that tableRow to json

Answer 1

Sounds like the provided example should work. Please make sure that you are running p.run() , as it is important to actually run the pipeline.

Here is a full example (using a public BigQuery dataset) that has worked for me:

  public static void main(String[] args) {
    PipelineOptions pipelineOptions = PipelineOptionsFactory.fromArgs(args).create();

    Pipeline pipeline = Pipeline.create(pipelineOptions);
    pipeline
        .apply(
            "ReadSourceBQ",
            BigQueryIO.readTableRows()
                .fromQuery("SELECT gameId FROM `bigquery-public-data`.`baseball`.`games_post_wide`")
                .usingStandardSql()
                .withTemplateCompatibility())
        .apply("JSON Transform", AsJsons.of(TableRow.class))
        .apply(
            "WriteToPubSub",
            PubsubIO.writeStrings().to("projects/{project_id}/topics/{topic_name}"));

    pipeline.run();
  }

Also, please note that AsJsons comes from beam-sdks-java-extensions-json-jackson , so you might have to import that using the following:

Maven

<dependency>
    <groupId>org.apache.beam</groupId>
    <artifactId>beam-sdks-java-extensions-json-jackson</artifactId>
    <version>{beam_version}</version>
</dependency>

Gradle

implementation group: 'org.apache.beam', name: 'beam-sdks-java-extensions-json-jackson', version: '{beam_version}'