如何在 Databricks/Spark 中读取 Google BigQuery 查看数据

Question

How can I read Google BigQuery view data in spark? I am able to read data from Google BigQuery tables using Service Account Authentication by using below query

df = (spark.read.format("bigquery").option("credentialsFile",credentialfilepath)
      .option("parentProject",projectName)
      .option("table",ViewName)
      .load())

But I am not able to read view data by using this, also I have applies other options as well like .option("viewsEnabled", "true") and .option("materializationDataset","dataplatform") but It is giving the error:

com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.BigQueryException: Access Denied: Project projectname: User does not have bigquery.jobs.create permission in project projectname

Can anyone suggest the way to read view data?

Answer 1

Providing the proper roles for the service account used for authentication solved the problem. See steps below on adding the roles to the service account used for authentication:

1. Open Google Cloud Console

2. Go to IAM & Admin > IAM. Look for the service account to be used.

3. When a service account is identified, click the edit button (pencil icon) at the right side.

4. Click "+ Add Another Role" and add the following roles

   • BigQuery Data editor
   • BigQuery Job User
   • BigQuery Data Viewer
   • BigQuery Read Session User
   • Storage Object Admin

   

5. Click Save

6. Run the spark code provided in the question