[英]How to append different PubSub objects and flatten them to write them altogether into bigquery as a single JSON?
I wanted to write three attributes (data, attributes and publish time) of a Pub/Sub message to Bigquery and wanted them to print in a flattened way so that all elements writes in a single row, for example:我想将 Pub/Sub 消息的三个属性(数据、属性和发布时间)写入 Bigquery,并希望它们以扁平化的方式打印,以便所有元素写入一行,例如:
data[0]![]() |
data[1]![]() |
attr[0]![]() |
attr[0]![]() |
key![]() |
publishTime![]() |
---|---|---|---|---|---|
data![]() |
data![]() |
attr![]() |
attr![]() |
key![]() |
publishTime![]() |
I'm currently using the following piece of code for decoding and parsing the message but this is applicable only for the data part of the Pub/Sub message:我目前正在使用以下代码来解码和解析消息,但这仅适用于 Pub/Sub 消息的数据部分:
class decodeMessage: def decode_base64(self,element): """Decode base64, padding being optional.""" return json.dumps(element.data.decode("utf-8")) class parseMessage: def parseJsonMessage(self,element): return(json.loads(element))
I've also tried merging two json after dumping them from Json objects to Json string but it didn't go as planned, my ultimate goal is to bring all columns into a single JSON with the schema retained.在将两个 json 从 Json 对象转储到 Json 字符串后,我也尝试合并它们,但它没有按计划将 go 合并,我的最终目标是将所有列合并为一个 JSON 并保留架构。
I hope my question remains clear to you!我希望你能清楚我的问题! Thanks!
谢谢!
The solution to the following problem is to simply make a Python dictionary and append all the data into a new Dictionary.下面这个问题的解决方法就是简单的把一个Python的字典和append的所有数据做成一个新的Dictionary。
example:例子:
payload = dict()
data = json.dumps(element.data.decode('utf-8'))
attributes = json.dumps(element.attributes)
messageKey = element.message_id
publish_time = (element.publish_time).timestamp()*1000
payload['et'] = publish_time
payload['data'] = data
payload['attributes'] = attributes
payload['key'] = messageKey
return (payload)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.