Spark中RDD的映射方法

Question

I'm trying to do stuff with Spark (in Scala). I have a Transformer class that looks like this:

class Transformer(transformerParameters: TransformerParameters) {
  // Process the parameters

  def transform(element: String): String = {
    // Do stuff
  }
}

I would like to do something like

val originalRDD = sc.textFile("blah")
val transformer = new Transformer(parameters)
val transformedRDD = originalRDD.map(transformer.transform)

Assuming I don't want to or can't make the Transformer class serializable, and further assuming that TransformerParameters is in fact serializable, I've seen people suggest writing instead (or I may have misunderstood):

val transformedRDD = originalRDD.map(new Transformer(parameters).transform)

I'm fine with creating a new Transformer instance on each JVM of the cluster, but it looks like this creates a new Transformer for every line, which seems unnecessary and potentially very expensive. Is this actually what it does? Is there a way not to make a new instance for every line?

Thanks!

Answer 1

You could broadcast (implicitly or explicitly) an object that has a field for the parameters, as well as a transient field reference to a Transformer.

You have a method on this object that delegates to transform on the Transformer, but first does lazy initialisation of the Transformer (check to see if the Transformer reference is initialised, if not create one with the parameters, then call transform).

In the map method you then call wrapper.transform rather than Transformer.transform - this saves the object creation on each call, and works around the serialization problem as each task gets it's own wrapper instance and hence it's own Transformer that will get reused.