如何將pyspark或sql中的1行4列dataframe轉換為4行2列dataframe

Question

我有一個 dataframe 返回 output 作為

我想將其轉換為

有人可以幫助了解如何准備 pyspark 代碼以動態實現此結果。 我在 sql 嘗試過 Unpivot 但沒有成功。

Answer 1

df =spark.createDataFrame([
(78,20,19,90),
],
('Machines',  'Books',  'Vehicles', 'Plants'))

創建一個新的 struct column 數組，其中包含列名稱和值名稱。 使用魔法內聯來分解結構字段。 下面的代碼

df.withColumn('tab', F.array(*[F.struct(lit(x).alias('Fields'), col(x).alias('Count')).alias(x) for x in df.columns])).selectExpr('inline(tab)').show()


+--------+-----+
|  Fields|Count|
+--------+-----+
|Machines|   78|
|   Books|   20|
|Vehicles|   19|
|  Plants|   90|
+--------+-----+

Answer 2

如unpivot-dataframe tutorial use 中所述：

df = df.selectExpr("""stack(4, "Machines", Machines, "Books", Books, "Vehicles", Vehicles, "Plants", Plants) as (Fields, Count)""")

或者概括一下：

cols = [f'"{c}", {c}' for c in df.columns]
exprs = f"stack({len(cols)}, {', '.join(str(c) for c in cols)}) as (Fields, Count)"
df = df.selectExpr(exprs)

完整示例：

df = spark.createDataFrame(data=[[78,20,19,90]], schema=['Machines','Books','Vehicles','Plants'])

# Hard coded
# df = df.selectExpr("""stack(4, "Machines", Machines, "Books", Books, "Vehicles", Vehicles, "Plants", Plants) as (Fields, Count)""")

# Generalised
cols = [f'"{c}", {c}' for c in df.columns]
exprs = f"stack({len(cols)}, {', '.join(str(c) for c in cols)}) as (Fields, Count)"
df = df.selectExpr(exprs)

[Out]:
+--------+-----+
|Fields  |Count|
+--------+-----+
|Machines|78   |
|Books   |20   |
|Vehicles|19   |
|Plants  |90   |
+--------+-----+