How can I connect to two different projects in Google BigQuery in Python?

Question

I'm trying to connect to two different BigQuery projects in Python, but I'm not sure how to do it. I have two JSON keys, both of which work when run separately, but I want to run them together so I can do a join from two different projects.

Here is my code:

credentials = service_account.Credentials.from_service_account_file('C:\\Users\\zuser\\Desktop\\JSON_Keys\\firstjsonkey.json')
credentials_2ndproject = service_account.Credentials.from_service_account_file('C:\\Users\\zuser\\Desktop\\JSON_Keys\\2ndjsonkey.json')

client = bigquery.Client(credentials=credentials, project="project1")
client_2ndproject = bigquery.Client(credentials=credentials_2ndproject, project="project2")

This code is accepted.

Here is where there is a problem:

sales = client.query(sql).to_dataframe()

I know this will bring in one project, but how do I bring in the second if my sql query has a join?

Thanks in advance!

Answer 1

write a SQL query with joins from both tables from both the projects. Execute it as a single query. You need not to create different big query clients instead

sameple_query = f"""select * from {{project1}}.{{dataset1}}.{{table1}} left join select * from {{project2}}.{{dataset2}}.{{table2}}"""

1. use a service account with permissions on both projects to create the credentials object and big query client creation.

credentials = service_account.Credentials.from_service_account_file('service_account_key_for_both_projects_bigquery_access.json')
bqclient = bigquery.Client(credentials=credentials)
bqclient.query(sample_query).result().to_dataframe()

Note: result() will wait for the query to complete and then return the dataframe.

Or

2. If your user has access to both projects, you can use the same query with bigquery client from any of the project.

# getting the credentials and project details for gcp project
    credentials, your_project_id = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
# creating a big query clients
    bqclient = bigquery.Client(
        credentials=credentials,
        project=your_project_id
    )