[英]Kryo Serialization not registering even after registering the class in conf
I made a class Person and registered it but on runtime, it shows class not registered.Why is it showing so?我创建了一个类 Person 并注册了它,但在运行时,它显示类未注册。为什么会这样显示?
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Failed to serialize task 0, not attempting to retry it. Exception during serialization: java.io.IOException: java.lang.IllegalArgumentException: Class is not registered: KyroExample$Person[]
Note: To register this class use: kryo.register(KyroExample$Person[].class);
Here is the sample code:这是示例代码:
val conf = new SparkConf().setAppName("kyroExample").setMaster("local")
conf.set("spark.serializer","org.apache.spark.serializer.KryoSerializer")
conf.registerKryoClasses(Array(classOf[Person],classOf[String])) //registered the class
conf.set("spark.kryo.registrationRequired", "true")
val sparkContext = new SparkContext(conf)
case class Person(name:String, age:Int) //this is the class
val personList: immutable.Seq[Person] = (1 to 100000).map(value=> Person(value+"",value))
val rdd: RDD[Person] = sparkContext.parallelize(personList)
val evenAge: RDD[Person] = rdd.filter(_.age %2 ==0)
evenAge.persist(StorageLevel.MEMORY_ONLY_SER)
evenAge.count()
evenAge.persist(StorageLevel.MEMORY_ONLY_SER)
evenAge.count()
Thread.sleep(200000)
It worked after registering both use cases:它在注册两个用例后起作用:
.registerKryoClasses(
Array(classOf[Person],classOf[Array[Person]])
)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.