LabeledPoint类的Kryo注册

Question

I am trying to run a very simple scala class in spark with Kryo registration. This class just loads data from a file into an RDD[LabeledPoint] .

The code (inspired from the one in https://spark.apache.org/docs/latest/mllib-decision-tree.html ):

import org.apache.spark.{SparkContext, SparkConf}

import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.regression.LabeledPoint



object test {
  def main(args: Array[String]) {

    val conf = new SparkConf().setMaster("local").setAppName("test")
    conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
    conf.set("spark.kryo.registrationRequired", "true")
    val sc = new SparkContext(conf)

    sc.getConf.registerKryoClasses(classOf[ org.apache.spark.mllib.regression.LabeledPoint ])
    sc.getConf.registerKryoClasses(classOf[ org.apache.spark.rdd.RDD[org.apache.spark.mllib.regression.LabeledPoint] ])

    // Load data
    val rawData = sc.textFile("data/mllib/sample_tree_data.csv")
    val data = rawData.map { line =>
      val parts = line.split(',').map(_.toDouble)
      LabeledPoint(parts(0), Vectors.dense(parts.tail))
    }

    sc.stop()
    System.exit(0)
  }
}

What I understand i that, as I have set spark.kryo.registrationRequired = true , all utilized classes must be registered, so that I have registered RDD[LabeledPoint] and LabeledPoint .

The problem

I receive the following error:

java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.mllib.regression.LabeledPoint[]
Note: To register this class use: kryo.register(org.apache.spark.mllib.regression.LabeledPoint[].class);
    at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442)
    at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
    at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472)
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:565)
    at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:162)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

As I understand it, it means that the class LabeledPoint[] is not registered, whereas I have registered the class LabeledPoint .

Furthermore, the code proposed in the error to register the class ( kryo.register(org.apache.spark.mllib.regression.LabeledPoint[].class); ) does not work.

• What is the difference between the two classes?
• How can I register this class?

Answer 1

Thanks a lot to @eliasah who contributed a lot to this answer by pointing out that the proposed solution ( kryo.register(org.apache.spark.mllib.regression.LabeledPoint[].class); ) is in Java and not in Scala.

Hence, what LabeledPoint[] means in Scala is Array[LabeledPoint] .

I solved my problem by registering the Array[LabeledPoint] class, ie adding in my code:

sc.getConf.registerKryoClasses(classOf[ Array[org.apache.spark.mllib.regression.LabeledPoint] ])