[英]Google BigQuery: In Python, column addition makes all the other columns Nullable
我有一个已经存在的表,其架构如下:
{
"schema": {
"fields": [
{
"mode": "required",
"name": "full_name",
"type": "string"
},
{
"mode": "required",
"name": "age",
"type": "integer"
}]
}
}
它已经包含以下条目:
{'full_name': 'John Doe',
'age': int(33)}
我想插入带有新字段的新记录,并让加载作业在加载时自动添加新列。 新格式如下所示:
record = {'full_name': 'Karen Walker',
'age': int(48),
'zipcode': '63021'}
我的代码如下:
from google.cloud import bigquery
client = bigquery.Client(project=projectname)
table = client.get_table(table_id)
config = bigquery.LoadJobConfig()
config.autoedetect = True
config.source_format = bigquery.SourceFormat.NEWLINE_DELIMITED_JSON
config.write_disposition = bigquery.WriteDisposition.WRITE_APPEND
config.schema_update_options = [
bigquery.SchemaUpdateOption.ALLOW_FIELD_ADDITION,
]
job = client.load_table_from_json([record], table, job_config=config)
job.result()
这会导致以下错误:
400 提供的架构与表 my_project:my_dataset:mytable 不匹配。 字段年龄已将模式从 REQUIRED 更改为 NULLABLE
我可以通过更改config.schema_update_options
来解决这个问题,如下所示:
bigquery.SchemaUpdateOption.ALLOW_FIELD_ADDITION,
bigquery.SchemaUpdateOption.ALLOW_FIELD_RELAXATION
]
这允许我插入新记录,并将zipcode
添加到架构中,但它会导致full_name
和age
都变为NULLABLE
,这不是我想要的行为。 有没有办法防止模式自动检测更改现有列?
如果您需要向架构中添加字段,您可以执行以下操作:
from google.cloud import bigquery
client = bigquery.Client()
table = client.get_table("your-project.your-dataset.your-table")
original_schema = table.schema # Get your current table's schema
new_schema = original_schema[:] # Creates a copy of the schema.
# Add new field to schema
new_schema.append(bigquery.SchemaField("new_field", "STRING"))
# Set new schema in your table object
table.schema = new_schema
# Call API to update your table with the new schema
table = client.update_table(table, ["schema"])
更新表的架构后,您可以使用此附加字段加载新记录,而忽略任何架构配置。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.