如何使用 pyspark 展平列表中的嵌套 json

Question

I have multiple nested jsons in list list=[nestes_json1,nested_json2,nested_json3]

I need to take each nested_json and remove comma at the end and process it and load into single dataframe. how to do it using pyspark.

I'm able to flatten nested_json individually but not in the form of list.

Answer 1

lets suppose you are having this type of table

Then you can try the explode function

from pyspark.sql.functions import explode
df.select(df.name,explode(df.subjects)).show(truncateenter =False)

and then you can get the following results

After this you can explode that new table again and you will get the exact formatting