扁平化 pyspark 嵌套结构 - Pyspark

Question

I want to flatten a nested column into separate ones, containing only a few specific values.

First off, I start with an AVRO capture file from Event hub. Once I convert that to a dataframe in python, I receive the following column after removing the irrelevant ones.

This column has the following structure.

What I want to do next, is flatten this column remaining specific values.

I can get this done for one cell, but because I am dealing with a nested structure the column is not iterable.

Anyone can help me out?

Answer 1

You have 2 data fields in your schema so I'd call the first one data_struct and the second one data_array . If you do avro_df_body.select('data_struct.data_array') , you'd have an ArrayType column, which you can apply explode function to break it down to multiple rows.

(avro_df_body
    .withColumn('tmp', F.explode('data_struct.data_array'))
    .withColumn('speed', F.col('tmp.a'))
    .withColumn('timestamp', F.col('tmp.time'))
    .show()
)