![](/img/trans.png)
[英]How to use BigQuery Storage API to concurrently read streams in Python threads
[英]How to use nested proto.Message with BigQuery Storage API Writer python client?
基于来自https://github.com/googleapis/python-bigquery-storage/issues/398的代码段,它使用proto-plus
package 来定义 python 中的 protobuff 消息,非常有帮助并且按原样运行良好,但以防万一嵌套消息的它不起作用。
以下改编代码会引发错误: google.api_core.exceptions.InvalidArgument: 400 Invalid proto schema: BqMessage.proto: Message.nested: "._default_package.Team" is not defined.
如果消息是嵌套的,则调用await bq_write_client.append_rows(iter([append_row_request]))
时。
PS 我知道google-cloud-bigquery-storag
库通常与嵌套消息一起使用,因为使用官方代码段https://github.com/googleapis/python-bigquery-storage/blob/main/samples/snippets/append_rows_proto2 .py有效,它使用嵌套消息,但在单独的 .proto 文件中,该文件需要编译步骤,不如直接在 python 中定义消息实用。
# Copyright 2021 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import json
import asyncio
import proto
from google.oauth2.service_account import Credentials
from google.protobuf.descriptor_pb2 import DescriptorProto
from google.cloud.bigquery_storage_v1beta2.types.storage import AppendRowsRequest
from google.cloud.bigquery_storage_v1beta2.types.protobuf import ProtoSchema, ProtoRows
from google.cloud.bigquery_storage_v1beta2.services.big_query_write import BigQueryWriteAsyncClient
class Team(proto.Message):
name = proto.Field(proto.STRING, number=1)
class UserSchema(proto.Message):
username = proto.Field(proto.STRING, number=1)
email = proto.Field(proto.STRING, number=2)
team = proto.Field(Team, number=3)
async def main():
write_stream_path = BigQueryWriteAsyncClient.write_stream_path(
"yolocommon", "test", "t_test_data", "_default")
credentials = Credentials.from_service_account_file(filename="bigquery_config_file.json")
bq_write_client = BigQueryWriteAsyncClient(credentials=credentials)
proto_descriptor = DescriptorProto()
UserSchema.pb().DESCRIPTOR.CopyToProto(proto_descriptor)
proto_schema = ProtoSchema(proto_descriptor=proto_descriptor)
serialized_rows = []
data = [
{
"username": "Jack",
"email": "jack@google.com",
"nested": {
"name": "Jack Jack"
}
},
{
"username": "mary",
"email": "mary@google.com",
"nested": {
"name": "Mary Mary"
}
}
]
for item in data:
instance = UserSchema.from_json(payload=json.dumps(item))
serialized_rows.append(UserSchema.serialize(instance))
proto_data = AppendRowsRequest.ProtoData(
rows=ProtoRows(serialized_rows=serialized_rows),
writer_schema=proto_schema
)
append_row_request = AppendRowsRequest(
write_stream=write_stream_path,
proto_rows=proto_data
)
result = await bq_write_client.append_rows(iter([append_row_request]))
async for item in result:
print(item)
if __name__ == "__main__":
asyncio.run(main())
更新:来自ProtoSchema
的文档:
输入消息的描述符。 提供的描述符必须是自包含的,这样发送的数据行就可以仅使用单个描述符进行完全解码。 对于由多个独立消息组成的数据行,这意味着描述符可能需要转换为仅使用嵌套类型: https://developers.google.com/protocol-buffers/docs/proto#nested所以正确的方法是写消息的描述是:
class UserSchema(proto.Message):
class Team(proto.Message):
name = proto.Field(proto.STRING, number=1)
username = proto.Field(proto.STRING, number=1)
email = proto.Field(proto.STRING, number=2)
team = proto.Field(Team, number=3)
但它仍然会抛出相同的错误: google.api_core.exceptions.InvalidArgument: 400 Invalid proto schema: BqMessage.proto: Message.nested: "._default_package.UserSchema.Team" is not defined.
UPDATE2:问题的根源是如果 package 名称为空, proto-plus
会将_default_package
附加为 package 名称,因为这会导致另一个错误。 https://github.com/googleapis/proto-plus-python/blob/main/proto/_package_info.py#L40
TODO:在 protobuf 修复后恢复为空字符串作为 package 值。 当 package 为空时,基于 upb 的 protobuf 在尝试添加到描述符池期间失败并显示“TypeError:无法将 proto 文件构建到描述符池:无效名称:空部分 ()' means”。
显然,目前不可能使用proto.Message
来表示 BigQuery 表,如果它有一个嵌套字段 (STRUCT)。
protobuf 已修复,因此 fork 项目并更改行: https://github.com/googleapis/proto-plus-python/blob/main/proto/_package_info.py#L40
到
package = getattr(
proto_module, "package", module_name if module_name else ""
)
它会起作用
以下模块有助于绕过 proto-plus 中的预编译或消息 class 定义。 https://pypi.org/project/xia-easy-proto/1.0.0/
您可以将 python object 解析并转换为 protobuff。 希望它能有所帮助。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.