简体   繁体   中英

How to register Receiver[] with Kryo?

In the process of porting a Spark 1.6 app to Spark 2.0.2, there's this complaint in the log:

com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.streaming.receiver.Receiver[]
Note: To register this class use: kryo.register(org.apache.spark.streaming.receiver.Receiver[].class);

This fails with Caused by: java.lang.ClassNotFoundException: org/apache/spark/streaming/receiver/Receiver[]/class :

sparkConf.set("spark.kryo.classesToRegister", "org.apache.spark.streaming.receiver.Receiver[].class")

This fails with Caused by: java.lang.ClassNotFoundException: org/apache/spark/streaming/receiver/Receiver[] :

sparkConf.set("spark.kryo.classesToRegister", "org.apache.spark.streaming.receiver.Receiver[]")

This fails with Class is not registered: org.apache.spark.streaming.receiver.Receiver[] :

sparkConf.set("spark.kryo.classesToRegister", "org.apache.spark.streaming.receiver.Receiver")

This fails with Class is not registered: org.apache.spark.streaming.receiver.Receiver[] :

sparkConf.registerKryoClasses(Array(
    classOf[org.apache.spark.streaming.receiver.Receiver[_]]
))

How can I get this class registered? I've been able to register other classes with Kryo, but not this one.

Edit:

In all these cases this setup is done:

sparkConf.set("spark.kryo.registrationRequired", "true")
sparkConf.set("spark.serializer", classOf[KryoSerializer].getName)
GraphXUtils.registerKryoClasses(sparkConf)

Found an approach that works here: Kryo serialization refuses to register class

In short, change classOf[org.apache.spark.streaming.receiver.Receiver[_]], to classOf[Array[org.apache.spark.streaming.receiver.Receiver[_]]],

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM