[英]Saving json from aws-glue into postgres, jsonb type column
[英]AWS Glue denest postgres jsonb column
我想將jsonb列展平為同一表中的多個目標列。 我找不到內置函數來完成此任務。 Glue搜尋器將jsonb列注冊為字符串。 當我將數據放到s3上時,可以使用Unbox.apply()將其更改為結構。
我曾嘗試使用Relationalize和UnnestFrame解密jsonb列。 都不起作用。 Relationalize似乎僅應用go .json文件。 我不確定UnnestFrame為什么不起作用。
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
## @params: [JOB_NAME]
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "mycatalogdb", table_name = "sourcedb_public_tablename", transformation_ctx = "datasource0")
dfc = UnnestFrame.apply(frame = datasource0, transformation_ctx = "dfc", info="", stageThreshold=0, totalThreshold=0)
dropnullfields3 = DropNullFields.apply(frame = dfc, transformation_ctx = "dropnullfields3")
datasink4 = glueContext.write_dynamic_frame.from_options(frame = dropnullfields3, connection_type = "s3", connection_options = {"path": "s3://mybucket"}, format = "parquet", transformation_ctx = "datasink4")
job.commit()
給定具有以下內容的源表
+----+------------+-------------------------------------------------------+
| id | date | myjson |
+----+------------+-------------------------------------------------------+
| 1 | 2019-10-10 | {"url":some-url,"data":{"afield":123,"moredata":567"} |
+----+------------+-------------------------------------------------------+
我想要此輸出(列名格式與表格格式無關緊要)
+----+------------+----------+-------------+---------------+
| id | date | url | data_afield | data_moredata |
+----+------------+----------+-------------+---------------+
| 1 | 2019-10-10 | some-url | 123 | 567 |
+----+------------+----------+-------------+---------------+
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.