使用通配符选项从 BigQuery 导出到 GCS 时，有没有办法从大表中获取生成的文件列表？

Question

I used the wildcard * export in order to export a large BigQuery table into separate files in GCS. I used the code sample provided in GCP's docs:

from google.cloud import bigquery
client = bigquery.Client()
bucket_name = 'bucket'
project = "project"
dataset_id = "dataset"
table_id = "table"


destination_uri = "gs://{}/{}".format(bucket_name, "table*.parquet")
dataset_ref = bigquery.DatasetReference(project, dataset_id)
table_ref = dataset_ref.table(table_id)

extract_job = client.extract_table(
    table_ref,
    destination_uri,
    # Location must match that of the source table.
    location="US",
)  # API request
extract_job.result()  # Waits for job to complete.

print(
    "Exported {}:{}.{} to {}".format(project, dataset_id, table_id, destination_uri)
)

This generated 19 different files in my storage bucket like this mytable000000000000.parquet and mytable000000000001.parquet and so on (up to 0000000000019).

It would be nice to have an automatic way to get a list of these file names so that I can either compose them together or loop over them to do something else. Is there an easy way to edit the code above to do this?

Answer 1

You don't get an explicit list when using a wildcard, but take a look at the destinationUriFileCounts field in the extract job statistics . It would tell you how many files are present. In python, this is available here .

If you want stronger validation, you could also leverage the Cloud Storage libraries and list objects with the same pattern(s) you supplied as part of the extract configuration.