使用通配符和自动检测将数据从GCS加载到BigQuery

Question

New to posting on StackOverflow.

Using the google.cloud.bigquery python SDK, I have been trying to work up a solution to load data from GCS to BigQuery without defining a table schema.

My LoadJobConfig 's autodetect is set to True and I am using a wildcard (*) in the GCS URI.

I have confirmed that Autodetect works with wildcards but the load job fails because the data source that I am working with will usually autodetect a specific column to be a float (eg 0.30) but sometimes adds operator symbols (eg < 0.10) and thus needs to be a string.

Can anyone think of a solution without having to define the schema? Here's my LoadJobConfig that I've passed to bigquery.client.Client 's load_table_from_uri method.

source_uri = 'gs://%s/%s/%s/*' % (source, report_type, date)
job_config = bigquery.LoadJobConfig()
job_config.create_disposition = 'CREATE_IF_NEEDED'
job_config.skip_leading_rows = 1
job_config.source_format = 'CSV'
job_config.write_disposition = 'WRITE_TRUNCATE'
job_config.autodetect = True
job = bigquery_client.load_table_from_uri(source_uri, table_ref, job_config=job_config)
job.result()

Answer 1

Your data seems to be broken in some part.

I would suggest to use flag: --max_bad_records which skips broken records.

For details please look here: https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-csv#bigquery-import-gcs-file-python