无法在本地运行 GCP DataflowTemplates

Question

我正在尝试使用https://github.com/GoogleCloudPlatform/DataflowTemplates和直接运行器在本地运行 PubSubToBigQuery.java。 但是我收到错误消息

Exception in thread "main" java.lang.IllegalArgumentException: Class interface com.google.cloud.teleport.templates.PubSubToBigQuery$Options missing a property named 'gcs-location'.
    at org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1518)
    at org.apache.beam.sdk.options.PipelineOptionsFactory.access$400(PipelineOptionsFactory.java:111)
    at org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.as(PipelineOptionsFactory.java:294)
    at com.google.cloud.teleport.templates.PubSubToBigQuery.main(PubSubToBigQuery.java:165)

但是我已经在运行期间通过了--gcs-location=gs://xxx-templates/dataflow/pipelines/pubsub-to-bigquery 。

正是在这一行，错误被抛出。 https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/main/java/com/google/cloud/teleport/templates/PubSubToBigQuery.java#L176

https://beam.apache.org/documentation/runners/direct/

Answer 1

您将传递给 Java 应用程序的 args 与传递给通过 CLI 运行模板化管道的 args 混淆了。

--gcs-location是您传递给在 CLI 上gcloud dataflow jobs run的内容。 当您运行 Java 应用程序时，Dataflow 会在 GCS（模板）上暂存您的管道，但不会立即运行管道。 --gcs-location告诉gcloud dataflow..模板运行的位置。

您不能在本地执行模板化管道。 您只需通过 Java 应用程序在本地运行模板的暂存。

https://cloud.google.com/dataflow/docs/guides/templates/executing-templates

 * # Set the runner
 * RUNNER=DataflowRunner
 *
 * # Build the template <--NOTE THIS
 * mvn compile exec:java \
 * -Dexec.mainClass=com.google.cloud.teleport.templates.PubSubToBigQuery \
 * -Dexec.cleanupDaemonThreads=false \
 * -Dexec.args=" \
 * --project=${PROJECT_ID} \
 * --stagingLocation=${PIPELINE_FOLDER}/staging \
 * --tempLocation=${PIPELINE_FOLDER}/temp \
 * --templateLocation=${PIPELINE_FOLDER}/template \
 * --runner=${RUNNER}"
 *
 * # Execute the template <--NOTE THIS
 * JOB_NAME=pubsub-to-bigquery-$USER-`date +"%Y%m%d-%H%M%S%z"`
 *
 * gcloud dataflow jobs run ${JOB_NAME} \
 * --gcs-location=${PIPELINE_FOLDER}/template \
 * --zone=us-east1-d \
 * --parameters \
 * "inputTopic=projects/data-analytics-pocs/topics/teleport-pubsub-to-bigquery,\
 * outputTableSpec=data-analytics-pocs:demo.pubsub_to_bigquery,\
 * outputDeadletterTable=data-analytics-pocs:demo.pubsub_to_bigquery_deadletter"
 * </pre>
 */