How to use 3rd party libraries which are not Serializable with Spark

Question

I want to use a 3rd party library(Wiremock) with Spark. However, I get the following exception:

Caused by: java.io.NotSerializableException:com.github.tomakehurst.wiremock.WireMockServer
Serialization stack:
    - object not serializable(class:com.github.tomakehurst.wiremock.WireMockServer, value: com.github.tomakehurst.wiremock.WireMockServer@51813065)

Is there a general way to deal with this?

Answer 1

There are few options:

1. Kryo might be able to serialize these objects out of the box, depending what's inside them. Try turning it on as described at http://spark.apache.org/docs/latest/tuning.html
2. If that doesn't work, you can create your own “wrapper” objects that implement Serializable, or even a subclass of com.github.tomakehurst.wiremock.WireMockServer . No need to change the original library.
3. If the library has its own serialization functions, you could also use those inside a wrapper object. Take a look at https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SerializableWritable.scala for an example where Spark make Hadoop's Writables serializable.

Answer 2

You can make the field that is not serializable transient . Then it will just not be serialized.

private transient WireMockServer wireMockServer;