为什么这个 Spark 代码在本地模式下工作，而不在集群模式下工作？

Question

So, I have something like this. Note that baseTrait (a trait) here is serializable, and therefore thisClass (an Object class) should also be serializable.

object thisClass extends baseTrait {
  private var someVar = null 

  def someFunc: RDD[...] {
    ...
    // assigned some string value or an empty string value (not null anymore)
    someVar = ... 
    ...
    if (someVar != "")
      someRDD.filter(x => aFunc(x, someVar))
    else
      ...
  }

In cluster mode, when I call the someFunc function (which is a static method since thisClass is an Object class) I get a null pointer exception, which I think has to do with someVar not being serialized properly. Because when I do this, it works perfectly in cluster mode.

if (someVar != "") {
  val someVar_ = someVar
  someRDD.filter(x => aFunc(x, someVar_))
}

Any idea what was going wrong in the original code, when thisClass is serializable in the first place?

My guess is that its fine to use a variable of a serializable class from within another class, but if you try to do it inside that class, you can have problems, as in that case you would have the runtime trying to serialize the same class where the closure is being called from. What do you think?

Answer 1

You are not experiencing a problem with serialization in this case.

Basically, what happens in cluster mode is that thisClass.someFunc is never actually executed in the remote executor's JVM. On the executor, thisClass is instantiated, and someVar is assigned null . Then while the thisClass object is in that state, the spark framework executes your lambda function directly on the records that are available in that executor's partition of the data.

A way to avoid this is to move the assignment to someVar into the body of the thisClass object. Doing that will assign the value to someVar immediately when the object is instantiated. Bear in mind that this code will be executed on every executor in the cluster.

If that is not possible, another option would be to map your RDD[T] to RDD[(T, String)] , where the string is someVar for every record, and then your filter could be something like .filter(x => aFunc(x._1, x._2)) . This method will use more memory, as you'll have many copies of someVar 's value.