[英]Spark Kryo Registrator
I'm using kryo as serializer in spark 2.1 I have set registration as required but I have facing a problem, I can't register this class:我在 spark 2.1 中使用 kryo 作为序列化程序我已经根据需要设置了注册,但我遇到了一个问题,我无法注册这个类:
org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex$SerializableFileStatus$SerializableBlockLocation[]
I have tried with classOf[Array[Class.forName("org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex$SerializableFileStatus$SerializableBlockLocation").type]]
and other combinations but I don't succeed in the registration of this class.我已经尝试过
classOf[Array[Class.forName("org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex$SerializableFileStatus$SerializableBlockLocation").type]]
和其他组合,但我没有成功注册这个类。
I have tried as well creating a java class like register and registering the class as org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex$SerializableFileStatus$SerializableBlockLocation[].class;
我也尝试过创建一个 java 类,例如 register 并将该类注册为
org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex$SerializableFileStatus$SerializableBlockLocation[].class;
Any clue is appreciated, Thanks.任何线索表示赞赏,谢谢。
I have solved my question, I've used:我已经解决了我的问题,我用过:
kryo.register(ClassTag(Class.forName("org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex$SerializableBlockLocation")).wrap.runtimeClass) kryo.register(ClassTag(Class.forName("org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex$SerializableBlockLocation")).wrap.runtimeClass)
It allows register for the class name at runtime wrapped in an array.它允许在运行时注册包含在数组中的类名。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.