Deserialization Error in spark using immutable class with java (+ lombok)

Question

I have this simple model class

@Value // lombok - create standard all arg constructor and getters
public class ModelA implements Serializable {
    private String word;
    private double value;
}

And this simple test fails:

public class SparkSerializationTest {

    private SparkSession spark = SparkSession.builder()
            .master("local")
            .appName("Test")
            .getOrCreate();

    @Test
    public void testSerializationModelA() {
        ModelA   modelA1 = new ModelA("A1", 12.34);
        ModelA   modelA2 = new ModelA("A2", 56.78);

        Dataset<ModelA> dataset = spark.createDataset(
                Arrays.asList(modelA1, modelA2),
                Encoders.bean(ModelA.class));

        List<ModelA> yo = dataset.collectAsList(); // <== *** failure here ***

        assertThat(yo).isEqualTo(Arrays.asList(modelA1, modelA2));
    }
}

with exception:

java.util.concurrent.ExecutionException: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 24, Column 67: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 24, Column 67: No applicable constructor/method found for zero actual parameters; candidates are: "com.xxx.yyy.ModelA(java.lang.String, double)"

It seems it requires a zero arg constructor. But I want my model to be immutable, thus with a full arg constructor and no setter. How should I do this?

Answer 1

easy way out

Just give it no-args constructor without any setters. It will be mutable but in slightly less chaotic fashion than if you provided all the setters. When you use Kryo as your deserializer (which I think you do already) you can keep this constructor private.

All-args constructor still can be called with nulls and insensical values. If you want to impose some contract on validity of the object, use validation explicitly. If it is immutability you are after, your members will not be final anymore using no-arg constructor.

dynamic nature of object creation

The deserialization works by calling the simplest (no-args) constructor as this is much easier process to implement for the authors of utility used, rather than assembling one call to the all-args constructor with all the necessary properties order of which could be arbitrary and the assignment to object properties not guaranteed.

Instead they create the vanila object and populate it via setters or reflection making sure the names match between serialised and object versions. All-args constructor would do this less reliably and would be much harder to implement.

custom object creation in Kryo

If you need to keep your immutability, you have to use custom object creation. Please have a look at Kryo's example for custom object creation :

 Registration registration = kryo.register(SomeClass.class); registration.setInstantiator(new ObjectInstantiator<SomeClass>() { public SomeClass newInstance () { return new SomeClass("some constructor arguments", 1234); } });