Google BigQuery 数据传输 - 数据集复制

Question

I have a BigQuery dataset in my Google project project-everest:evr_dataset, I want to copy the its table data to my another BigQuery dataset which is sitting in an another project, project-alps:alp_dataset.

I attempted to use DTS - data transfer service to schedule the ingest job on a daily basis, but I don't see any option to choose the destination dataset from another project? Can anyone enlighten me how enable inter-project DTS?

Answer 1

You can use the below python function to create big query data transfer client and copy datsets from one project to another by specifying source and target project id's. You can also schedule the data transfer. In the method below it is set to 24 hours(daily).

def copy_dataset(override_values={}):
    # [START bigquerydatatransfer_copy_dataset]
    from google.cloud import bigquery_datatransfer

    transfer_client = bigquery_datatransfer.DataTransferServiceClient()

    destination_project_id = "my-destination-project"
    destination_dataset_id = "my_destination_dataset"
    source_project_id = "my-source-project"
    source_dataset_id = "my_source_dataset"
    # [END bigquerydatatransfer_copy_dataset]
    # To facilitate testing, we replace values with alternatives
    # provided by the testing harness.
    destination_project_id = override_values.get(
        "destination_project_id", destination_project_id
    )
    destination_dataset_id = override_values.get(
        "destination_dataset_id", destination_dataset_id
    )
    source_project_id = override_values.get("source_project_id", source_project_id)
    source_dataset_id = override_values.get("source_dataset_id", source_dataset_id)
    # [START bigquerydatatransfer_copy_dataset]
    transfer_config = bigquery_datatransfer.TransferConfig(
        destination_dataset_id=destination_dataset_id,
        display_name="Your Dataset Copy Name",
        data_source_id="cross_region_copy",
        params={
            "source_project_id": source_project_id,
            "source_dataset_id": source_dataset_id,
        },
        schedule="every 24 hours",
    )
    transfer_config = transfer_client.create_transfer_config(
        parent=transfer_client.common_project_path(destination_project_id),
        transfer_config=transfer_config,
    )
    print(f"Created transfer config: {transfer_config.name}")
    # [END bigquerydatatransfer_copy_dataset]
    return transfer_config