Bigquery 在使用 Python 加载 json 文件时将我的字符串字段转换为 integer

Question

{"number":"1234123"} I am assigning this data to my Bigquery table using bigquery.LoadJobConfig in python. The type of my number column in my bigquery table is string. When I do the load operation, it converts the data type in my bigquery table to integer. How can I solve this? The file type I loaded: json.

job_config = bigquery.LoadJobConfig(
create_disposition=bigquery.CreateDisposition.CREATE_IF_NEEDED,
write_disposition=bigquery.WriteDisposition.WRITE_APPEND,
source_format=bigquery.SourceFormat.NEWLINE_DELIMITED_JSON,autodetect=True
)

Additionally: When I set autodetect to False, I get an error like Error while reading data, error message: JSON table encountered too many errors

Answer 1

I recommend you to pass a BigQuery schema to prevent this situation, instead to use autodetect=True , example:

from google.cloud import bigquery

# Construct a BigQuery client object.
client = bigquery.Client()

# TODO(developer): Set table_id to the ID of the table to create.
# table_id = "your-project.your_dataset.your_table_name"

job_config = bigquery.LoadJobConfig(
    schema=[
        bigquery.SchemaField("number", "STRING")
    ],
    create_disposition=bigquery.CreateDisposition.CREATE_IF_NEEDED,
    write_disposition=bigquery.WriteDisposition.WRITE_APPEND,
    source_format=bigquery.SourceFormat.NEWLINE_DELIMITED_JSON,
    autodetect=False
)
uri = "gs://cloud-samples-data/bigquery/us-states/us-states.json"

load_job = client.load_table_from_uri(
    uri,
    table_id,
    location="US",  # Must match the destination dataset location.
    job_config=job_config,
)  # Make an API request.

load_job.result()  # Waits for the job to complete.

destination_table = client.get_table(table_id)
print("Loaded {} rows.".format(destination_table.num_rows))

In this example I set the schema of BigQuery table and autodetect to False . If you use autodetect to True , you can't have a control on your field types.

You can check the documentation to have more information.