[英]Data transformation in Spark Scala
I have the following dataframe 我有以下数据框
+-----+-----+-----+ .......+-------+
|item1|item2|item3|........| itemN |
+-----+-----+-----+........|-------+
| v1| v2| v3|........| vN----+
| v4| v5| v6|........| v2N---+
+-----+-----+-----+........|-------+
here item1 , item2 and item3 are the column names and table consists of 1 row v1,v2,v3. 这里的item1,item2和item3是列名,表由1行v1,v2,v3组成。
I want to transform it into 我想把它变成
colA colB
item1 v1
item2 v2
item3 v3
. .
. .
. .
Here there are two columns lets say colA and colB and rows are as shown. 这里有两列,比如说colA和colB,行如图所示。
How to do this transformation in spark using scala? 如何使用Scala在Spark中进行此转换?
You can use explode
: 您可以使用explode
:
import org.apache.spark.sql.functions._
input.show()
// +-----+-----+-----+
// |item1|item2|item3|
// +-----+-----+-----+
// | v1| v2| v3|
// | v4| v5| v6|
// +-----+-----+-----+
val columns: Array[String] = input.columns
val result = input.explode(columns.map(s => col(s)): _*) {
r: Row => columns.zipWithIndex.map { case (name, index) => (name, r.getAs[String](index)) }
}.select($"_1" as "colA", $"_2" as "colB")
result.show()
// +-----+----+
// | colA|colB|
// +-----+----+
// |item1| v1|
// |item2| v2|
// |item3| v3|
// |item1| v4|
// |item2| v5|
// |item3| v6|
// +-----+----+
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.