如何从 gcs 存储桶中解压缩 tsv 文件并将其加载到 Bigquery

Question

Below is the code to get the tsv.gz file from gcs and unzip the file and converting into comma separated csv file to load csv data into Bigquery.

storage_client = storage.Client(project=project_id)
blobs_list = list(storage_client.list_blobs(bucket_name))
for blobs in blobs_list:
  if blobs.name.endswith(".tsv.gz"):
    source_file = blobs.name
    uri = "gs://{}/{}".format(bucket_name, source_file)
    gcs_file_system = gcsfs.GCSFileSystem(project=project_id)
    with gcs_file_system.open(uri) as f:
        gzf = gzip.GzipFile(mode="rb", fileobj=f)
        csv_table=pd.read_table(gzf)
        csv_table.to_csv('GfG.csv',index=False)

Code seems not effective to load data into BQ as getting many issues. Thought doing wrong with the conversion of file. Please put you thoughts where it went wrong?

Answer 1

If your file is gzip (not zip, I mean gzip), and in Cloud Storage, don't load it, unzip it and stream load it.

You can directly load, as is, in BigQuery, it's magic!! Here a sample

from google.cloud import bigquery

# Construct a BigQuery client object.
client = bigquery.Client()

# TODO(developer): Set table_id to the ID of the table to create.
# table_id = "your-project.your_dataset.your_table_name"

job_config = bigquery.LoadJobConfig(
    autodetect=True, #Automatic schema
    field_delimiter=",", # Use \t if your separator is tab in your TSV file
    skip_leading_rows=1, #Skip the header values(but keep it for the column naming)
    # The source format defaults to CSV, so the line below is optional.
    source_format=bigquery.SourceFormat.CSV,
)
uri = "gs://{}/{}".format(bucket_name, source_file)

load_job = client.load_table_from_uri(
    uri, table_id, job_config=job_config
)  # Make an API request.

load_job.result()  # Waits for the job to complete.