Spark Scala使用地图和过滤器读取文本文件

Question

I've a text file with following format ( id,f1,f2,f3,...,fn ):

12345,0,0,1,2,...,3
23456,0,0,1,2,...,0
33333,0,1,1,0,...,0
56789,1,0,0,0,...,4
a_123,0,0,0,6,...,3

And I want to read the file (ignore the line like a_123,0,0,0,6,...,3 ) to create a RDD[(Long, Vector) . Here's my solution:

  def readDataset(path: String, sparkSession: SparkSession): RDD[(ItemId, Vector)] = {
    val sc = sparkSession.sparkContext
    sc.textFile(path)
      .map({ line => val values=line.split(",")
        (
          values(0).toLong,
          //util.Try(values(0).toLong).getOrElse(0L),
          Vectors.dense(values.slice(1, values.length).map {x => x.toDouble }).toSparse
        )})
      .filter(x => x._1 > 0)
  }

However this code can not be compiled:

[ERROR]  found   : org.apache.spark.rdd.RDD[(Long, org.apache.spark.ml.linalg.SparseVector)]
[ERROR]  required: org.apache.spark.rdd.RDD[(Long, org.apache.spark.ml.linalg.Vector)]
[ERROR]     (which expands to)  org.apache.spark.rdd.RDD[(Long, org.apache.spark.ml.linalg.Vector)]
[ERROR] Note: (Long, org.apache.spark.ml.linalg.SparseVector) <: (Long, org.apache.spark.ml.linalg.Vector), but class RDD is invariant in type T.
[ERROR] You may wish to define T as +T instead. (SLS 4.5)
[ERROR]       .filter(x => x._1 > 0)
[ERROR]              ^
[ERROR] one error found

But if I remove the . toSparse . toSparse or .filter(x => x._1 > 0) this code can be compiled successfully.

Does someone know why and what should I do to fix it?

Also is there any better way to read the file to RDD with ignoring the non-numeric id lines?

Answer 1

The code compiles successfully if you remove toSparse because the type of your PairRDD is (ItemId, Vector) .

The org.apache.spark.ml.linalg.Vector class/type represent the Dense Vector which you are generating using Vector.dense and when you call toSparse it gets converted to org.apache.spark.ml.linalg.SparseVector which is not the type that your PairRDD expects.

As for filtering non-integer IDs I would say your method is a good way to do that.