在 RDD 上计算之前为每个 Spark 执行器指定初始化

Question

I want to make sure this class/static-block to be initialized in every JVM as early as possible because it suppose to be(in order to work properly),

// java
class SecurityInitializer {
    static {
        Security.addProvider(myprovider);
    }
}

// Equivalent in Scala, use companion object to get something similar to static block in Java
object SecurityInitializer {
    Security.addProvider(myprovider)
}

I know it`s easy to do this in driver , but how do I force this code to run in executor ideally before creation of sparkcontext?

Looking for answers in Scala but any suggestion is welcomed.

Answer 1

OK 现在是官方用例了，从 2.4 开始，名字是 ExecutorPlugin https://issues.apache.org/jira/browse/SPARK-24918