是否可以直接在Spark工作者中创建变量？

Question

What I would like to do is generate a context inside every Spark worker that I can use for local look-ups. The look-up data is located in a database and I would like to cache it on every worker. Is there a simple way to do this?

Workarounds used:

1. Create a lazily initialized Broadcast variable and use it with my functions. The first time a function tries to access it, I call my SQL code to initialize it.
2. Create an eagerly initialized Broadcast and use torrent broadcasting to make it available in workers

PS. I did not use JdbcRDD because I want the data to be replicated rather than partitioned. Does anyone knows what would happen if I did not use the partitioning attributes of the JdbcRDD ? Would that just make it work or would it have a non-deterministic behavior?

Answer 1

You could create a singleton object containing a reference to the resolution cache you want to use:

object ResolutionCache {
   var connection = _
   var cache: Map[Key,Value] = Map()
   def resolve(key:Key):Value = ???
}

Then this object can be used to resolve values in an RDD operation:

val resolved = keysRDD.map(key => (key -> ResolutionCache.resolve(key)))

The connections and values held by this object will be maintained independently per worker JVM. We must take special care of the connection management and concurrent behavior. In particular, resolve must be thread-safe.