![](/img/trans.png)
[英]org.apache.spark.mllib.linalg.Vector to DataFrame scala
[英]Convert an org.apache.spark.mllib.linalg.Vector RDD to a DataFrame in Spark using Scala
我有一个[Int Int Int]的org.apache.spark.mllib.linalg.Vector RDD。 我试图使用此代码将其转换为数据帧
import sqlContext.implicits._
import org.apache.spark.sql.types.StructType
import org.apache.spark.sql.types.StructField
import org.apache.spark.sql.types.DataTypes
import org.apache.spark.sql.types.ArrayData
vectrdd属于org.apache.spark.mllib.linalg.Vector类型
val vectarr = vectrdd.toArray()
case class RFM(Recency: Integer, Frequency: Integer, Monetary: Integer)
val df = vectarr.map { case Array(p0, p1, p2) => RFM(p0, p1, p2) }.toDF()
我收到以下错误
warning: fruitless type test: a value of type
org.apache.spark.mllib.linalg.Vector cannot also be a Array[T]
val df = vectarr.map { case Array(p0, p1, p2) => RFM(p0, p1, p2) }.toDF()
error: pattern type is incompatible with expected type;
found : Array[T]
required: org.apache.spark.mllib.linalg.Vector
val df = vectarr.map { case Array(p0, p1, p2) => RFM(p0, p1, p2) }.toDF()
我尝试的第二种方法是这样的
val vectarr=vectrdd.toArray().take(2)
case class RFM(Recency: String, Frequency: String, Monetary: String)
val df = vectrdd.map { case (t0, t1, t2) => RFM(p0, p1, p2) }.toDF()
我收到了这个错误
error: constructor cannot be instantiated to expected type;
found : (T1, T2, T3)
required: org.apache.spark.mllib.linalg.Vector
val df = vectrdd.map { case (t0, t1, t2) => RFM(p0, p1, p2) }.toDF()
我使用此示例作为指南>> 在Spark / Scala中将RDD转换为Dataframe
vectarr
将具有Array[org.apache.spark.mllib.linalg.Vector]
类型,因此在模式匹配中,您无法匹配Array(p0, p1, p2)
因为匹配的是Vector,而不是Array。
此外,您不应该执行val vectarr = vectrdd.toArray()
- 这会将RDD转换为Array,然后最终调用toDF
将无法工作,因为toDF
仅适用于RDD。
正确的行是(假设您将RFM
更改为具有双打)
val df = vectrdd.map(_.toArray).map { case Array(p0, p1, p2) => RFM(p0, p1, p2)}.toDF()
或者,等效地用val arrayRDD = vectrdd.map(_.toArray())
替换val vectarr = vectrdd.toArray()
(生成Array[Vector]
val arrayRDD = vectrdd.map(_.toArray())
(生成RDD[Array[Double]]
)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.