Pyspark - 将结构为 arrays 的列扩展到新列中

Question

我有一个 DataFrame，它有一个单列，它是一个结构类型并包含一个数组。

users_tp_df.printSchema()
root
 |-- x: struct (nullable = true)
 |    |-- ActiveDirectoryName: string (nullable = true)
 |    |-- AvailableFrom: string (nullable = true)
 |    |-- AvailableFutureAllocation: long (nullable = true)
 |    |-- AvailableFutureHours: double (nullable = true)
 |    |-- CreateDate: string (nullable = true)
 |    |-- CurrentAllocation: long (nullable = true)
 |    |-- CurrentAvailableHours: double (nullable = true)
 |    |-- CustomFields: array (nullable = true)
 |    |    |-- element: struct (containsNull = true)
 |    |    |    |-- Name: string (nullable = true)
 |    |    |    |-- Type: string (nullable = true)
 |    |    |    |-- Value: string (nullable = true)

我正在尝试将 CustomFields 数组列转换为 3 个三列：

1. 国家;
2. 是外部的；
3. 服务。

因此，例如，我有这些值：

并且除该行之外的最终 dataframe output 将是：

谁能帮我实现这个目标？

谢谢！

Answer 1

考虑到下面的模型结构，与您示例中的模型结构类似，您可以使用内联 function 以 sql 的方式进行：

with alpha as (
select named_struct("alpha", "abc", "beta", 2.5, "gamma", 3, "delta"
                    , array(  named_struct("a", "x", "b", "y", "c", "z")
                            , named_struct("a", "xx", "b", "yy", "c","zz"))
                   ) root
)
select root.alpha, root.beta, root.gamma, inline(root.delta) as (a, b, c) 
from alpha

结果：

模型结构：

Answer 2

这会起作用：

initial_expansion= df.withColumn("id", F.monotonically_increasing_id()).select("id","x.*");

final_df = initial_expansion\
     .join(initial_expansion.withColumn("CustomFields", F.explode("CustomFields"))\
           .select("*", "CustomFields.*")\
           .groupBy("id").pivot("Name").agg(F.first("Value")), \
        "id").drop("CustomFields")

示例输入：Json - {'x': {'CurrentAvailableHours': 2, 'CustomFields': [{'Name': 'Country', 'Value': 'Italy'}, {'Name': 'Service', 'Value':'Dev'}]}}

输入结构：

root
 |-- x: struct (nullable = true)
 |    |-- CurrentAvailableHours: integer (nullable = true)
 |    |-- CustomFields: array (nullable = true)
 |    |    |-- element: struct (containsNull = true)
 |    |    |    |-- Name: string (nullable = true)
 |    |    |    |-- Value: string (nullable = true)

Output：

Output 结构体（Id可以去掉）：

root
 |-- id: long (nullable = false)
 |-- CurrentAvailableHours: integer (nullable = true)
 |-- Country: string (nullable = true)
 |-- Service: string (nullable = true)