如何运行用 Golang 编写的 GCP Cloud Function 以运行数据流作业以将文本文件导入 Spanner？

Question

I have used the example in: https://github.com/apache/beam/blob/master/sdks/go/examples/wordcount/wordcount.go#L82 as well as the advice from Google Cloud Support to use the following to run a Dataflow import job:

    flag.Parse()
    flag.Set("runner", "dataflow")
    flag.Set("project"xxxx "rp")
    flag.Set("region", "us-central1")
    flag.Set("staging_location", "gs://xxx/temp")
    flag.Set("job_name", "import-delivery_schedule")
    beam.Init()
    p := beam.NewPipeline()
    p.Root()
    if err := beamx.Run(context.Background(), p); err != nil {
        log.Fatalf("Failed to execute job: %v", err)
    }

The cloud function is in the same project as the database. The Dataflow import job is in the same project as the database. The import job runs successfully from the console.

However, I'm unable to get this to work.

I'm getting this error: "Function execution took 18 ms, finished with status: 'connection error'"

Answer 1

I'm open to other ways to import a text file into a spanner table. What would you suggest?

Answer 2

If the import dataflow job is getting created and running successfully, then there seems to be no problem with the GCP cloud function.

Ensure the dataflow workers have sufficient permissions to access the database. https://cloud.google.com/spanner/docs/import#iam These permissions are required for a dataflow job to access and write to spanner (or a subset of them depending on the type of modifications you are doing in the import). Include these roles to the service account which the dataflow workers are assuming. https://cloud.google.com/dataflow/docs/concepts/security-and-permissions#worker-service-account