[英]How to convert scala.collection.Set to java.util.Set with serializable within an RDD
I have a scala.collection.Set scalaSet : Set[Long]
. 我有一个scala.collection.Set
scalaSet : Set[Long]
。
How will I be able to convert it into a java.util.Set
with serializable. 我怎样才能将它转换为带序列化的
java.util.Set
。 I tried the following code, but got java.io.notserializableexception: scala.collection.convert.wrappers$setWrapper
我尝试了以下代码,但得到了
java.io.notserializableexception: scala.collection.convert.wrappers$setWrapper
import scala.collection.JavaConversions._
Class MySerializableClass extends Serializable {
// method to implement the Scala to Java operations on the given RDD
def rddOps(dummyRDD: RDD[(Long, Set[Long])]) = {
val dummyRDDWithJavaSet = dummyRDD.map( {
case(key, value) => (key, scalaToJavaSetConverter(value))
}
// scala Set to Java Set Converters
def scalaToJavaSetConverter(scalaSet: Set[Long]): java.util.Set[Long] = {
val javaSet : java.util.Set[Long] = setAsJavaSet(scalaSet)
javaSet
}
}
I have seen the thread notserializable exception when trying to serialize java map converted from scala for an answer, but the solution didn't work with serialization 我在尝试序列化从scala转换的java地图时看到了线程不可序列化的异常 ,但是该解决方案不适用于序列化
The serialization issue with the scala.collection.JavaConvertions/JavaConverters
is that these converters are wrappers that use the underlying (scala/java) object. scala.collection.JavaConvertions/JavaConverters
的序列化问题是这些转换器是使用底层(scala / java)对象的包装器。 They are merely a wrapper and therefore for it to be effectively serializable, they must have a warranty that the underlying structure is serializable. 它们只是一个包装器,因此它可以有效地序列化,它们必须保证底层结构是可序列化的。
The easiest solution in your case is to implement a structural copy in your conversion method: 在您的情况下,最简单的解决方案是在转换方法中实现结构副本:
// scala Set to Java Set Converters
def scalaToJavaSetConverter(scalaSet: Set[Long]): java.util.Set[Long] = {
val javaSet = new java.util.HashSet[Long]()
scalaSet.foreach(entry => javaSet.add(entry))
javaSet
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.