使用python将Google存储器中的csv.gz文件加载到bigquery

Question

I want to load csv.gz file from storage to bigquery. Right now I using below code, but I am not sure if it is efficient way to load data to bigquery.

# -*- coding: utf-8 -*-
from io import BytesIO
import pandas as pd
from google.cloud import storage
import pandas_gbq as gbq
client = storage.Client.from_service_account_json(service_account)
bucket = client.get_bucket("bucketname")
blob = storage.blob.Blob("""somefile.csv.gz""", bucket)
content = blob.download_as_string()
df = pd.read_csv(BytesIO(content), delimiter=',', quotechar='"', low_memory=False)
df = df.astype(str)
df.columns = df.columns.str.replace("|", "")
df["dateinsert"] = pd.datetime.now()
gbq.to_gbq(df, 'desttable',
           'projectid',
           chunksize=None,
           if_exists='append'
           )

Please assist me to write this code in efficient way

Answer 1

I propose you this process:

• Perform a load job into bigquery
  • Add the schema, yes 150 column is boring...
  • Add skip leading row option for skipping the header job_config.skip_leading_rows = 1
  • Name your table like this <dataset>.<tableBaseName>_<Datetime> The date time must be a string format compliant with BigQuery table name. For example YYYYMMDDHHMM

When you query your data, you can query a subset of table, and inject the table name in the query result, like this:

SELECT *,(SELECT table_id
      FROM `<project>.<dataset>.__TABLES_SUMMARY__`
      WHERE table_id LIKE '<tableBaseName>%') FROM `<project>.<dataset>.<tableBaseName>*` 

Of course, you can raffine the * with the year, month, day,...

I think, I meet all your requirements. Comment if something goes wrong