来自org.apache.spark.rdd.RDD [（（（Any，Any），Iterable [org.apache.spark.sql.Row]）的Spark Sql数据

Question

我有org.apache.spark.rdd.RDD[((Any, Any), Iterable[org.apache.spark.sql.Row])]数据，
如何打印数据或获取数据？

我有这样的代码：

val sessionsDF = Seq(("day1","user1","session1", 100.0),
  ("day1","user1","session2",200.0),
  ("day2","user1","session3",300.0),
  ("day2","user1","session4",400.0),
  ("day2","user1","session4",99.0)
).toDF("day","userId","sessionId","purchaseTotal").toDF()

val groupByData=sessionsDF.groupBy(x=>(x.get(0),x.get(1))
val filterData=groupByData.filter(x=>x._1._1=="day1").map(x=>x._2)

上面的代码返回org.apache.spark.rdd.RDD[((Any, Any), Iterable[org.apache.spark.sql.Row])]

Answer 1

第一步，您需要额外的.toDF() 。 正确的一个如下

val sessionsDF = Seq(("day1","user1","session1", 100.0),
  ("day1","user1","session2",200.0),
  ("day2","user1","session3",300.0),
  ("day2","user1","session4",400.0),
  ("day2","user1","session4",99.0)
).toDF("day","userId","sessionId","purchaseTotal")

在第二步中，您错过了.rdd因此实际的第二步是

val groupByData=sessionsDF.rdd.groupBy(x=>(x.get(0),x.get(1)))

如您在问题中提到的那样具有dataType

scala> groupByData
res12: org.apache.spark.rdd.RDD[((Any, Any), Iterable[org.apache.spark.sql.Row])] = ShuffledRDD[9] at groupBy at <console>:25

要查看groupByData rdd您可以简单地使用foreach作为

groupByData.foreach(println)

这会给你

((day1,user1),CompactBuffer([day1,user1,session1,100.0], [day1,user1,session2,200.0]))
((day2,user1),CompactBuffer([day2,user1,session3,300.0], [day2,user1,session4,400.0], [day2,user1,session4,99.0]))

现在，您的第三步是过滤将day1值作为day column值的dataframe 。 而且，您仅采用分组的 rdd数据的值。

val filterData=groupByData.filter(x=>x._1._1=="day1").map(x=>x._2)

此步骤返回的dataType是

scala> filterData
res13: org.apache.spark.rdd.RDD[Iterable[org.apache.spark.sql.Row]] = MapPartitionsRDD[11] at map at <console>:27

您可以使用上面的foreach来查看数据

filterData.foreach(println)

这会给你

CompactBuffer([day1,user1,session1,100.0], [day1,user1,session2,200.0])

您可以看到返回的dataType是RDD[Iterable[org.apache.spark.sql.Row]]因此您可以使用map将每个值打印为

filterData.map(x => x.map(y => println(y(0), y(1), y(2), y(3)))).collect

这会给你

(day1,user1,session1,100.0)
(day1,user1,session2,200.0)

如果你只做

filterData.map(x => x.map(y => println(y(0), y(3)))).collect

你会得到

(day1,100.0)
(day1,200.0)

我希望答案是有帮助的