执行程序上的Spark对象（单例）序列化

Question

I am not sure that what I want to achieve is possible. What I do know is I am accessing a singleton object from an executor to ensure it's constructor has been called only once on each executor. This pattern is already proven and works as expected for similar use cases in my code base.

However, What I would like to know is if I can ship the object after it has been initialized on the driver. In this scenario, when accesing ExecutorAccessedObject.y , ideally it would not call the println but just return the value. This is a highly simplified version, in reality, I would like to make a call to some external system on the driver, so when accessed on the executor, it will not re-call that external system. I am ok with @transient lazy val x to be reinitialized once on the executors, as that will hold a connection pool which cannot be serialized.

object ExecutorAccessedObject extends Serializable {
  @transient lazy val x: Int = {
    println("Ok with initialzing this on the executor. I.E. database connection pool")
    1
  }

  val y: Int = {
    // call some external system to return a value.
    // I do not want to call the external system from the executor
    println(
      """
        |Idealy, this would not be printed on the executor.
        |return value 1 without re initializing
      """)
    1
  }
  println("The constructor will be initialized Once on each executor")
}


someRdd.mapPartitions { part =>
  ExecutorAccessedObject
  ExecutorAccessedObject.x // first time accessed should re-evaluate
  ExecutorAccessedObject.y // idealy, never re-evaluate and return 1
  part
}

I attempted to solve this with broadcast variables as well, but I am unsure how to access the broadcast variable within the singleton object.

Answer 1

What I would like to know is if I can ship the object after it has been initialized on the driver.

You cannot. Objects , as singletons, are never shipped to executors. There initialized locally, whenever objects is accessed for the first time.

If the result of the call is serializable, just pass it alone, either as an arguments to the ExecutorAccessedObject (implicitly or explicitly) or making ExecutorAccessedObject mutable (and adding required synchronization).