pySpark：如何在 dataframe 的 arrayType 列中获取 structType 中的所有元素名称？

Question

I have a dataframe that looks something like this:

 |-- name: string (nullable = true)
 |-- age: string (nullable = true)
 |-- job: string (nullable = true)
 |-- hobbies: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- favorite: string (nullable = true)
 |    |    |-- non-favorite: string (nullable = true)

And I'm trying to get this information:

['favorite', 'non-favorite']

However, the only closest solution I found was using the explode function with withColumn , but it was based on the assumption that I already know the names of the elements. But What I want to do is, without knowing the element names, I want to get the element names only with the column name, in this case 'hobbies'. Is there a good way to get all the element names in any given column?

Answer 1

For a given dataframe with this schema:

df.printSchema()

root
 |-- hobbies: array (nullable = false)
 |    |-- element: struct (containsNull = false)
 |    |    |-- favorite: string (nullable = false)
 |    |    |-- non-favorite: string (nullable = false)

You can select the field names of the struct as:

struct_fields = df.schema['hobbies'].dataType.elementType.fieldNames()

# output: ['favorite', 'non-favorite']

Answer 2

pyspark.sql.types.StructType.fieldnames should get you what you want.

 fieldNames() Returns all field names in a list. >>> struct = StructType([StructField("f1", StringType(), True)]) >>> struct.fieldNames() ['f1']

So in your case something like

dataframe.hobbies.getItem(0).fieldnames()