组合 AWS Glue 作业中的字段

Question

My goal is to Extract record from a DDB table and transform them into an S3 JSON object.

The record in the DDB table are:

accounts displayname name objectname objectid objectdefinition

The S3 Object required definition is:

{
   "accounts": "example",
   "name": "exampleDisplay",
   "objectJson": "{
"displayname":"exampleDisplay",
"objectname":"exampleObjectName",
"objectId":"exampleObjectId",
"objectDefinition":"exampleDefintion",
}
}

Now the spark transform script to convert it into camelCase is easy. But how do I create a new field like objectJson and add certain fields from ddb to it as a JSON.

Answer 1

You can do this in following way

df = spark.createDataFrame([('accounts','name','displayname','objectname','objectId','objectDefinition'),('accounts','name','displayname','objectname','objectId','objectDefinition')], ['accounts','name','displayname','objectname','objectId','objectDefinition'])

df2 = df.withColumn('objectJson',f.struct(
                            f.col('displayname'),
                            f.col('objectname'),
                            f.col('objectId'),
                            f.col('objectDefinition')
)).select('accounts','name','objectJson')

df2.toJSON().take(100)

[{"accounts":"accounts",
   "name":"name",
   "objectJson":{
                 "displayname":"displayname",
                 "objectname":"objectname",
                 "objectId":"objectId",
                  objectDefinition":"objectDefinition"
}}]

df2.repartition(1).write.json('path/to/save/data')

Hope it helps