[英]Unpivot dataframe in Pyspark with new column
I would like to unpivot a dataframe that looks like this:我想取消旋转看起来像这样的 dataframe:
Col1 Col2 Val1 Val2
abc def 12 75
ghi jkl 67 86
... ... .. ..
into something that will look like this:变成这样的东西:
Col1 Col2 NewCol Val
abc def KEY1 12
abc def KEY2 75
ghi jkl KEY1 67
ghi jkl KEY2 86
... ... .... ..
I am quite new to python, but I know there is no unpivot function in pyspark.. any idea how I can achieve this?我对 python 很陌生,但我知道 pyspark 中没有 unpivot function .. 知道我怎么能做到这一点吗? Thanks a lot!
非常感谢!
Given the Dataframe you provided, one could use:鉴于您提供的 Dataframe,可以使用:
from pyspark.sql import functions as F
df.select(
F.col("Col1"),
F.col("Col2"),
F.explode(
F.map_from_arrays(
F.array(F.lit("key1"), F.lit("key2")),
F.array(F.col("val1"), F.col("val2"))
)
)
)
As long as you maintain the order of keys and values, you should be fine只要您保持键和值的顺序,就可以了
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.