Spark局部变量广播到执行器

Question

var countryMap = Map("Amy" -> "Canada", "Sam" -> "US", "Bob" -> "Canada")
val names = List("Amy", "Sam", "Eric")
sc.parallelize(names).flatMap(broadcastMap.value.get).collect.foreach(println)

//output
Canada
US

I'm running this spark job in YARN mode, and I'm sure that driver and executors are not in the same node/JVM (see the attached pic). Since countryMap is not a broadcast variable, so the executor should not see it and this code shouldn't print anything. However, it printed Canada and US .

My question is that does Spark populate local variables to executors automatically if they are serializable? if not, how does the executor see the driver's local variables?

Answer 1

Hay Edwards,

when you invoke collect that bring result set back to driver try to perform mapping. that the reason that you could find mappings get generated.

Cheers,

Answer 2

local variables: The driver and each executor,no serialization required，Shared within the actuator/driver. main variables: The driver and each copy(insulate) of the task,need serialization

Answer 3

Spark actions are executed through a set of stages, separated by distributed “shuffle” operations. Spark automatically broadcasts the common data needed by tasks within each stage. The data broadcasted this way is cached in serialized form and deserialized before running each task. This means that explicitly creating broadcast variables is only useful when tasks across multiple stages need the same data or when caching the data in deserialized form is important.

reference https://spark.apache.org/docs/2.3.0/rdd-programming-guide.html#broadcast-variables

Basically, local variables in the driver will be broadcast to executors automatically. However, you need to create broadcast variables when you need them across different stages.

Answer 4

To have broadcastMap.value.get function running on a cluster, Spark needs to serialize broadcastMap and send to every executor, so you have a function with data already attached to it in the form an object instance. If you make broadcastMap class unserializable - you won't be able to run this code whatsoever.

So, Spark doesn't populate local variables to executors, but rather you explicitly tell it to serialize object broadcastMap and run a method of that object distributedly.