[英]Converting List of List or RDD to DataFrame in Spark-Scala
所以基本上我想要實現的是-我有一個包含4列的表(例如),並將其公開給DataFrame-DF1。 現在,我想將DF1的每一行存儲到另一個配置單元表(基本上是DF2,其架構為-Column1,Column2,Column3),而column3的值將是DataFrame DF1的'-'分隔行。
val df = hiveContext.sql("from hive_table SELECT *")
val writeToHiveDf = df.filter(new Column("id").isNotNull)
var builder : List[(String, String, String)] = Nil
var finalOne = new ListBuffer[List[(String, String, String)]]()
writeToHiveDf.rdd.collect().foreach {
row =>
val item = row.mkString("-@")
builder = List(List("dummy", "NEVER_NULL_CONSTRAINT", "some alpha")).map{case List(a,b,c) => (a,b,c)}
finalOne += builder
}
現在,我將finalOne作為列表列表,我想直接將其或通過RDD轉換為數據框 。
var listRDD = sc.parallelize(finalOne) //Converts to RDD - It works.
val dataFrameForHive : DataFrame = listRDD.toDF("table_name", "constraint_applied", "data") //Doesn't work
錯誤:
java.lang.ClassCastException: org.apache.spark.sql.types.ArrayType cannot be cast to org.apache.spark.sql.types.StructType
at org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.scala:414)
at org.apache.spark.sql.SQLImplicits.rddToDataFrameHolder(SQLImplicits.scala:94)
有人可以幫我理解將其轉換為DataFrame的正確方法嗎? 提前感謝您的支持。
如果要在數據幀中使用3列類型的字符串,則應將List[List[(String,String,String)]]
展平為List[(String,String,String)]
:
var listRDD = sc.parallelize(finalOne.flatten) // makes List[(String,String,String)]
val dataFrameForHive : DataFrame = listRDD.toDF("table_name", "constraint_applied", "data")
我相信,在將“ finalOne”數據幀傳遞到sc.parallelize()函數之前,先將其展平應該會產生符合您期望的結果。
var listRDD = sc.parallelize(finalOne)
val dataFrameForHive : DataFrame = listRDD.toDF("table_name", "constraint_applied", "data")
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.