Scala中的Java对象序列化

Question

Pardon me as I am new to Scala. I have created a case class which encapsultes some information. One of the objects i want to take in for that is of JavaClass. As i am using in spark, i would need it to be serializable. How can i do that?

Java class

public class Currency {

    public Currency(final BigDecimal amount, final CurrencyUnit unit) {
        //Doing Something
    }
}

case class ReconEntity(inputCurrency : Currency, outputCurrency : Currency)

Using implicit i want to have my serialization code for Currency so that spark can work on ReconEntity.

Answer 1

Firstly, have you tried some RDD operations using your Currency and ReconEntity classes? Do you actually get an error? Spark is able to handle RDD operations with apparently non-serializable Scala classes as values, at least (you can try this in the spark-shell , though possibly this might require the Kryo serializer to be enabled).

Since you state that you don't own the Currency class, you can't add extends Serializable , which would be the simplest solution.

Another approach is to wrap the class with a serializable wrapper, as described in this article: Beating Serialization in Spark - example code copied here for convenience:

For simple classes, it is easiest to make a wrapper interface that extends Serializable. This means that even though UnserializableObject cannot be serialized we can pass in the following object without any issue

public interface UnserializableWrapper extends Serializable {
  public UnserializableObject create(String prm1, String prm2);
}

The object can then be passed into an RDD or Map function using the following approach

UnserializableWrapper usw = new UnserializableWrapper() {
  public UnserializableObject create(String prm1, String prm2) {
    return new UnserializableObject(prm1,prm2);
} }

If the class is merely a data structure, without significant methods, then it might be easier to unpack its fields into your RDD types (in your case, ReconEntity ) and discard the class itself.

If the class has methods that you need, then your other (ugly) option is to cut-and-paste code into a new serializable class or into helper functions in your Spark code.