Pyspark 數據框列包含字典數組，希望將字典中的每個鍵變成一列

Question

我目前有一個這樣的數據框：

+-------+-------+-------+-------+
| Id    |value_list_of_dicts    |
+-------+-------+-------+-------+
| 1     |[{"val1":0, "val2":0}, |
|       |{"val1":2, "val2":5}]  |
+-------+-------+-------+-------+
| 2     |[{"val1":9, "val2":10},|
|       |{"val1":1, "val2":2}]  |
+-------+-------+-------+-------+

每個列表恰好包含 30 個字典，值可能不同但鍵名始終相同。 我希望我的數據框看起來像這樣：

+-------+-------+-------+
| Id    |val1   |val2   |
+-------+-------+-------+
| 1     | 0     | 0     |
+-------+-------+-------+
| 1     | 2     | 5     |
+-------+-------+-------+
| 2     | 9     | 10    |
+-------+-------+-------+
| 2     | 1     | 2     |
+-------+-------+-------+

做這個的最好方式是什么？

Answer 1

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

from pyspark.sql.types import *
from datetime import datetime
from pyspark.sql import *
from collections import *
from pyspark.sql.functions import udf,explode
from pyspark.sql.types import StringType
df= spark.createDataFrame(
    [
        (1, [{"val1":0, "val2":0},{"val1":2, "val2":5}]), 
        (2, [{"val1":9, "val2":10},{"val1":1, "val2":2}])
       
        ],("ID","List")
    )

df2 = df.select(df.ID,explode(df.List).alias("Column1") )
df2.withColumn("Val1", F.col("Column1").getItem("val1")).withColumn("Val2", F.col("Column1").getItem("val2")).show(truncate=False)

輸出：

+---+-----------------------+----+----+
|ID |Column1                |Val1|Val2|
+---+-----------------------+----+----+
|1  |[val2 -> 0, val1 -> 0] |0   |0   |
|1  |[val2 -> 5, val1 -> 2] |2   |5   |
|2  |[val2 -> 10, val1 -> 9]|9   |10  |
|2  |[val2 -> 2, val1 -> 1] |1   |2   |

+---+-----------------------+----+----+