将来自云存储的外部临时表与预先存在的大查询表相结合 - append 来自 python

Question

I have a permanent table in bigquery that I want to append to with data coming from a csv in google cloud storage. I first read the csv file into a big query temp table:

table_id = "incremental_custs"
external_config = bigquery.ExternalConfig("CSV")
external_config.source_uris = [
    "gs://location/to/csv/customers_5083983446185_test.csv"
]
external_config.schema=schema
external_config.options.skip_leading_rows = 1
job_config = bigquery.QueryJobConfig(table_definitions={table_id: external_config})
sql_test = "SELECT * FROM `{table_id}`;".format(table_id=table_id)
query_job = bq_client.query(sql_test,job_config=job_config)
customer_updates = query_job.result()
print(customer_updates.total_rows)

Up until here all works and I retrieve the records from the tmp table. Issue arises when I try to then combine it with a permanent table:

sql = """
create table `{project_id}.{dataset}.{table_new}` as (
      select customer_id, email, accepts_marketing, first_name, last_name,phone,updated_at,orders_count,state,
              total_spent,last_order_name,tags,ll_email,points_approved,points_spent,guest,enrolled_at,ll_updated_at,referral_id,
              referred_by,referral_url,loyalty_tier_membership,insights_segment,rewards_claimed
              from (
                select * from `{project_id}.{dataset}.{old_table}`
                union all 
                select * from `{table_id}`
                ORDER BY customer_id, orders_count  DESC
                ))
                order by orders_count desc 
""".format(project_id=project_id, dataset=dataset_id, table_new=table_new, old_table=old_table, table_id=table_id)
query_job = bq_client.query(sql)
query_result = query_job.result()

I get the following error:

BadRequest: 400 Table name "incremental_custs" missing dataset while no default dataset is set in the request.

Am I missing something here? Thanks !

Answer 1

Arf, you forgot the external config! You don't pass it in your second script

query_job = bq_client.query(sql)

Simply update it like in the first one

query_job = bq_client.query(sql_test,job_config=job_config)

A fresh look is always easier!