![](/img/trans.png)
[英]How to perform basic statistics on a csv file to explore my numeric and non-numeric variables with Spark Scala?
[英]Pivot non-numeric table in Spark Scala
是否可以在Spark Scala中旋轉具有非數字值的表? 我已經審查了以下兩個Stack問題。
按照“何時列出案例”問題中的步驟進行操作,我可以轉換數據,以便每個數據類型都是一列,但是每個實體-數據類型組合都有一行。
id tag value
1 US foo
1 UK bar
1 CA baz
2 US hoo
2 UK hah
2 CA wah
id US UK CA
1 foo
1 bar
1 baz
2 hoo
2 hah
3 wah
是否有一個“第一個非空”函數可以將每個實體的多行折疊為一個?
id US UK CA
1 foo bar baz
2 hoo hah wah
這是一個完整的Scala類,它創建一個示例數據框,然后對其進行透視。 它是專門針對此問題的,因此我不知道它通常會有多大用處。 還沒有經過廣泛的測試,所以買家要當心。
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.sql.{Column, DataFrame, Row, SQLContext}
import org.apache.spark.sql.functions.{lit, when}
import org.apache.spark.sql.types.{IntegerType, StringType, StructField, StructType}
object DemoPivot {
def main(args: Array[String]) = {
def pivotColumn(df: DataFrame)(t: String): Column = {
val col = when(df("tag") <=> lit(t), df("value"))
col.alias(t)
}
def pivotFrame(sqlContext: SQLContext, df: DataFrame): DataFrame = {
val tags = df.select("tag").distinct.map(r => r.getString(0)).collect.toList
df.select(df("id") :: tags.map(pivotColumn(df)): _*)
}
def aggregateRows(value: Seq[Option[Any]], agg: Seq[Option[Any]]): Seq[Option[Any]] = {
for (i <- 0 until Math.max(value.size, agg.size)) yield i match {
case x if x > value.size => agg(x)
case y if y > agg.size => value(y)
case z if value(z).isEmpty => agg(z)
case a => value(a)
}
}
def collapseRows(sqlContext: SQLContext, df: DataFrame): DataFrame = {
// RDDs cannot have null elements, so pack into Options and unpack before returning
val rdd = df.map(row => (Some(row(0)), row.toSeq.tail.map(element => if (element == null) None else Some(element))))
val agg = rdd.reduceByKey(aggregateRows)
val aggRdd = agg.map{ case (key, list) => Row.fromSeq((key.get) :: (list.map(element => if (element.isDefined) element.get else null)).toList) }
sqlContext.createDataFrame(aggRdd, df.schema)
}
val conf = new SparkConf().setAppName("Simple Pivot Demo")
val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)
val data = List((1, "US", "foo"), (1, "UK", "bar"), (1, "CA", "baz"),
(2, "US", "hoo"), (2, "UK", "hah"), (2, "CA", "wah"))
val rows = data.map(d => Row.fromSeq(d.productIterator.toList))
val fields = Array(StructField("id", IntegerType, nullable = false),
StructField("tag", StringType, nullable = false),
StructField("value", StringType, nullable = false))
val df = sqlContext.createDataFrame(sc.parallelize(rows), StructType(fields))
df.show()
val pivoted = pivotFrame(sqlContext, df)
pivoted.show()
val collapsed = collapseRows(sqlContext, pivoted)
collapsed.show()
}
}
您可以考慮使用aggregate
方法(或aggregateByKey
)。 您只需要編寫適當的函數即可在每個位置獲取非null元素。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.