使用 Scala 在 Spark 中使用广播变量的正确语法是什么？

Question

I want to use a broadcast variable in Spark with Scala. But I can't find enough help on how to use them. Say, I have an object of class A, which I normally would declare as follows in Scala.

val a = new A()

What would be the syntax of declaring it as a broadcast variable. And how would I call its methods?

Answer 1

If sc is a SparkContext , then val broadcasted = sc.broadcast(a) will broadcast a .
You can then access it with broadcasted.value .

Answer 2

Marth is right. You need also to destroy a broadcast variable by using sc.destroy(blocking) where blocking is a flag. I want to highlight that is recommended to avoid to broadcast small variables.