以编程方式检索AWS Glue动态框架字段名称和数据类型

Question

New to PySpark and AWS Glue. I need to retrieve fields names and data types to use them in a program. I could print the results of DynamicFrame.schema(), which appear to be a JSON-like structure. I tried json.loads to see if the structure can be converted to Python dictionary, so I could iterate through it, but got this error - TypeError: expected string or buffer ------------------------------*

AWS Glue documentation doesn't indicate the return type of schema(). Thank you.

Answer 1

You can use DynamicFrame.schema().field_map to get a dictionary of all the fields and their types:

 {'field_map': 
   {'PersonId': Field(PersonId, IntegerType({}), {}),
    'ChangeId': Field(ChangeId, LongType({}), {}),
    'CreateDate': Field(CreateDate, TimestampType({}), {}),)},
 'fields': 
    [Field(PersonId, IntegerType({}), {}),
    Field(ChangeId, LongType({}), {}),
    Field(CreateDate, TimestampType({}), {})],
    'properties'