使用 bq 命令在 GCP Bigquery 和 Cloud SQL (Mysql 5.7) 之间创建连接时出现错误

Question

I want to create a connection between Bigquery and Cloud SQL(MySQL 5.7) in a newly created project , (not in existing project, I can create the connection in existing project with no problem)

This is the command I used

bq mk --connection --connection_type='CLOUD_SQL' --properties='{"instanceId":"<PROJECT ID>:<REGION>:<MYSQL INSTANCE>","database":"<MY DATABASE>","type":"MYSQL"}' --connection_credential='{"username":"root", "password":"<PASSWORD>"}' --project_id=<PROJECT ID> --location=<REGION> <MYSQL INSTANCE>

but I got the error like below

BigQuery error in mk operation: Access Denied: URI:  services/bigqueryconnection.googleapis.com/projects/<PROJECT ID>:
APPLICATION_ERROR;google.api.serviceconsumermanagement.v1beta1/ServiceConsumerManagerV1Beta1.GenerateServiceIdentity;Permission denied to generate service identity for service
[bigqueryconnection.googleapis.com]
Details: [{
IAM{policy: 'serviceconsumermanagement_consumers-/000000555846b828/bigqueryconnection.googleapis.com/000000b742218216' resource: 'services/bigqueryconnection.googleapis.com/consumers/<PROJECT ID>'
permission: 'serviceconsumermanagement.consumers.generateServiceAccount'}
allowed: false auditlog: false cloudaudit: false
}]

...
...

After a lot of research, I think the problem is maybe I don't have a service acccount like this with a role called BigQuery Connection Service agent . because I have this kind of service account in another project.

service-<project id>@gcp-sa-bigqueryconnection.iam.gserviceaccount.com

I don't really know how this service account being created, maybe by enabling the Bigquery Connection API service, I will have a service account like above being created.

Although the bigquery Connection API has already being enabled, I disalbed it then enabled it again. but the service account still not being created.

Does anyone has any idea about this problem? Thank you so much.

update:

Sorry I forgot to say I already have the bigquery Admin/bigquery connection admin/cloud sql admin permissions

Answer 1

You need grant bigquery.admin access. Document link

gcloud projects add-iam-policy-binding project_id \
--member group/user:address \
--role roles/bigquery.admin

Example:

$ bq mk --connection --connection_type='CLOUD_SQL' --properties='{"instanceId":"data-lab:us-east1:instance-mysql","database":"information_schema","type":"MYSQL"}' --connection_credential='{"username":"root", "password":"*****"}' --project_id=data-lab --location=us MyExternalSQL
Connection 906721254566.us.MyExternalSQL successfully created

Answer 2

When you enable the BigQuery Connection API, a service account should be automatically created on your behalf with the following roles:

    cloudsql.client
    logging.logWriter
    metrics.metricWriter

and to create and maintain a connection resource, user must have the bigquery.admin predefined role as @Soumendra mentioned.

Check Document Cloud SQL federated queries

I enabled the API, but a service account wasn't created.

Check Document BigQuery Connection API Client Libraries for creating that service account.

Once that is done, try connecting.