用于 BigQuery 存储 API 上 Apache Beam 2.39.0 和 DataFlow 运行器的 PERMISSION_DENIED

Question

I have the following error for one of my DataFlow Jobs:

2022-06-15T16:12:27.365182607Z Error message from worker: java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: BigQuery Storage API has not been used in project 770406736630 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/bigquerystorage.googleapis.com/overview?project=770406736630 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

The same code works fine with Apache Beam 2.38.0. I tested multiple times and this is not a temporary issues. The project number mentioned in the error (770406736630) is not mine.

Any idea why I get this error?

Answer 1

I had the same issue. I'm using Spring Cloud GCP and hadn't set the spring.cloud.gcp.project-id property, which I'm guessing makes the SDK or API use some default value.

I don't know how you've set up you environment, because you haven't specified, but look into how you can explicitly set the project id. You can get it from the dialog for selecting a project in GCP Console.

Answer 2

I just ran into this, and simply needed to re-authenticate with the gcp cli by running gcloud auth application-default login .

Answer 3

The error happens for the latest Apache Beam SKD (2.41.0) when BigQueryIO.Write.Method.STORAGE_WRITE_API is used and destination does not specify the project name. For example dataset.table instead of project-id:dataset.table

This is the solution that worked for me:

    BigQueryIO.writeTableRows()
        .to("project-id:dataset.table")
        .withMethod(BigQueryIO.Write.Method.STORAGE_WRITE_API)

For some reason the Apache Beam implementation for BigQuery Write Storage API does not handle this situation even though it works fine for FILE_LOADS method.

You may also receive a sightly different error for the latest Beam SDK.

Exception in thread "main" org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.RuntimeException: 
java.lang.RuntimeException: 
java.lang.RuntimeException: com.google.api.gax.rpc.PermissionDeniedException:
io.grpc.StatusRuntimeException: 
PERMISSION_DENIED: Permission denied: Consumer 'project:null' has been suspended.