Connect to BigQuery with Service Account Credentials?

Question

I create a Google service account to access BigQuery, it provides a.json credentials file, which looks like:

{
  "type": "service_account",
  "project_id": "<redacted>",
  "private_key_id": "<redacted>",
  "private_key": "-----BEGIN PRIVATE KEY-----<redacted>\n-----END PRIVATE KEY-----\n",
  "client_email": "<redacted>",
  "client_id": "<redacted>",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "<redacted>"
}

Using a secrets manager, I load the above into a Python dict bigquery_credentials , then try to connect:

client = bigquery.Client(credentials=bigquery_credentials)

I get the error:

DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started

Clearly, I'm not passing the credentials in a valid way and bigquery.Client isn't recognizing them.

The docs say "Set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the path of the JSON file that contains your service account key."

I'd prefer to avoid using the file system. This code is running in an environment where I may not have access to the local file system. Also, I'm using a secret manager: storing secrets on the file system is considered a security risk.

Is there any way I can specify the service account.json credentials to bigquery.Client ?

Answer 1

I just had this one, you can use the following:

from google.cloud import bigquery

client = bigquery.Client.from_service_account_json("/path/to/your/file.json")