BigQuery 計划查詢 - 創建表並在不同項目中為其名稱添加日期后綴

Question

根據文檔，( https://cloud.google.com/bigquery/docs/scheduling-queries#destination_table ) 在為計划查詢定義目標時必須使用相同的項目。

但是，我想安排一個查詢，使其能夠在對其他項目的查詢步驟中寫入表（例如 CREATE TABLE xxx.dataset.name_{run_date} ）並保留 {run_date} 作為后綴。 在 BQ 中可以這樣做嗎？

Answer 1

這是BigQuery UI中的一個限制。 一種可能的解決方法是使用BigQuery Python客戶端庫，如下代碼所示：

from google.cloud import bigquery_datatransfer
from datetime import date

today = date.today() #use to replicate @run_date parameter
str_today = str(today).replace("-", "")

transfer_client = bigquery_datatransfer.DataTransferServiceClient()

# The project where the query job runs is the same as the project
# containing the destination dataset.
project_id = "your-project-id"
dataset_id = "your-source-dataset-id"

# This service account will be used to execute the scheduled queries. Omit
# this request parameter to run the query as the user with the credentials
# associated with this client.
service_account_name = "your-service-account"

# Use standard SQL syntax for the query.
query_string = f"CREATE TABLE destination-project.destination-dataset.new_table_{str_today} AS (SELECT column FROM source-project.source-dataset.source-table);"

parent="projects/your-project-id/locations/us-central1" #change location accordingly

transfer_config = bigquery_datatransfer.TransferConfig(
    name="projects/your-project-id/locations/us-central1/transferConfigs", #change location accordingly
    display_name="Test Schedule",
    data_source_id="scheduled_query",
    params={
        "query": query_string,
    },
    schedule="every 24 hours",
)

transfer_config = transfer_client.create_transfer_config(
    bigquery_datatransfer.CreateTransferConfigRequest(
        parent=parent,
        transfer_config=transfer_config,
        service_account_name=service_account_name,
    )
)

print("Created scheduled query '{}'".format(transfer_config.name))

我的示例輸出表：