New to PySpark and AWS Glue. I need to retrieve fields names and data types to use them in a program. I could print the results of DynamicFrame.schema(), which appear to be a JSON-like structure. I tried json.loads to see if the structure can be converted to Python dictionary, so I could iterate through it, but got this error - TypeError: expected string or buffer ------------------------------*
AWS Glue documentation doesn't indicate the return type of schema(). Thank you.
You can use DynamicFrame.schema().field_map
to get a dictionary of all the fields and their types:
{'field_map':
{'PersonId': Field(PersonId, IntegerType({}), {}),
'ChangeId': Field(ChangeId, LongType({}), {}),
'CreateDate': Field(CreateDate, TimestampType({}), {}),)},
'fields':
[Field(PersonId, IntegerType({}), {}),
Field(ChangeId, LongType({}), {}),
Field(CreateDate, TimestampType({}), {})],
'properties'
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.