Query from a BigQuery database via a Google Cloud Function (Python)

Question

I have a big Query Database connected to a Google Sheet in which I have a read only access My request is that I want to get data from a table and this request is working perfectly fine in the Big Query editor but I want to create a Google Cloud function to have an API and access this request directly from URL I have ceated a Service Account using this command:

gcloud iam service-accounts create connect-to-bigquery
gcloud projects add-iam-policy-binding test-24s --member="serviceAccount:connect-to-bigquery@test-24s.iam.gserviceaccount.com" --role="roles/owner"

and I have created a Google cloud function as follow: Creating Cloud Function

Service account settings

Here is my code for main.py file:

from google.cloud import bigquery
def hello_world(request):
 client = bigquery.Client()
 query = "SELECT order_id, MAX(status) AS last_status FROM `test-24s.Dataset.Order_Status` GROUP BY order_id"
 query_job = client.query(query)
 print("The query data:")
 for row in query_job:
   print("name={}, count ={}".format(row[0], row["last_status"]))
 return f'The query run successfully'

And for the requirements.txt file:

# Function dependencies, for example:
# package>=version

google-cloud-bigquery

The function deploys successfully however when I try to test it I get this error:

Error: function terminated. Recommended action: inspect logs for termination reason. Additional troubleshooting documentation can be found at https://cloud.google.com/functions/docs/troubleshooting#logging Details:
500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

And when reading the log file I found this error

403 Access Denied: BigQuery BigQuery: Permission denied while getting Drive credentials.

Please help me to solve this I already tried all the solutions that I found on the net without any success

Answer 1

Based on this: "Permission denied while getting Drive credentials" - I would say that your service account's IAM permissions are not 'transient' => while that service account probably has relevant access to the BigQuery, it does not have access to the underlined spreadsheet maintained on the Drive...

I would try - either

1. extend the scope of the service account's credentials (if possible, but that may not be very straightforward). Here is an article by Zaar Hai with some details - Google Auth — Dispelling the Magic and a comment from Guillaume - "Yes, my experience isn't the same" ;

or (preferably from my point of view)

2. make a copy (may be with regular updates) of the original spreadsheet based table as a native BigQuery table, and use the later in your cloud function. A side effect of this approach - a significant performance improvement (and cost savings).