Mercari Spanner 到 Bigquery - QUERY 中无法识别的参数

Question

Using the Mercari Tempalate for Spanner to BigQuery - https://github.com/mercari/DataflowTemplates

When building the dataflow through Google Console - it works.

But when executing the dataflow command through CLI, it generates an error "unrecognized arguments".

DATAFLOW COMMAND:

gcloud dataflow jobs run mercari_CLI --gcs-location gs://mystorage/templates/SpannerToBigQuery --region us-central1 --staging-location gs://mystorage/temp --parameters projectId=myProject,instanceId=myspanner,databaseId=mydpspanner,query=SELECT *, current_timestamp AS LoadDttm FROM source_table,output=raw_data_zone.testtable

Tried the following, but same error:

query="SELECT *, current_timestamp AS LoadDttm FROM source_table"
query="""SELECT *, current_timestamp AS LoadDttm FROM source_table"""
query='SELECT *, current_timestamp AS LoadDttm FROM source_table'
query=`SELECT *, current_timestamp AS LoadDttm FROM source_table`

Would anyone know any approach for this?

Answer 1

I guess you are looking for a escape sequence in gcloud command(since your job works when launching from gcloud console)

I would recommend going through "gcloud topic escaping" or the following link https://cloud.google.com/sdk/gcloud/reference/topic/escaping

There are examples to guide you in specifying a delimiter that is not present in your query.