使用python将bigquery分区替换为在bigquery表中暂存的数据

Question

I have 30 days of data staged in a daily partitioned table in bigquery. I have a larger table with 5 years of data partitioned daily. I need to select from the stage table and replace the entire contents of the existing partitions in the larger table for the 30 days that are in my staging table. My preference is to do this using Python and not extracting the data to a csv first and then loading it back to BQ if I can avoid that. Any suggestions? Thanks in advance.

Answer 1

All you need to do is query what you need and set destination table for your query.

from google.cloud import bigquery
client = bigquery.Client()
query = """\
SELECT firstname + ' ' + last_name AS full_name,
       FLOOR(DATEDIFF(CURRENT_DATE(), birth_date) / 365) AS age
 FROM dataset_name.persons
"""
dataset = client.dataset('dataset_name')
table = dataset.table(name='person_ages')
job = client.run_async_query('fullname-age-query-job', query)
job.destination = table
job.write_disposition= 'truncate'
job.begin()

Answer 2

That did not actually work for me but I do think it is correct, albeit for an older version of the big query client library. Your answer did help tremendously and I will accept. I am using most up to date library. The following worked for me:

for partition in gbq.list_partitions(stage_table_ref):
    table_partition = table_name+'$'+partition
    stage_partition = stage_dataset.table(table_partition)
    target_partition = target_dataset.table(table_partition)
    job_config = bigquery.CopyJobConfig()
    job_config.write_disposition = bigquery.WriteDisposition.WRITE_TRUNCATE   
    gbq.copy_table(stage_partition, target_partition,job_config = job_config)