Scala值未存储在spark中

Question

var cnt = 0
val newRDD = oldRDD.map({ list =>
        ...some work
        cnt = cnt + 1
        println(cnt) // print 1, 2, 3.. well
        newList // will store in new RDD
    })

//out side of map
println(cnt) // It's 0. Why?

As log, print inside of map is first and then print out side of map. Why isn't cnt value stored?

Answer 1

Spark transformations should be pure functions - receive input, produce output, without changing state or having any side-effects . Your example violates these.

What happens here is:

• The anonymous function passed as argument to map is serialized and sent to workers
• The initial value of cnt is serialized with it
• cnt is 0 when deserialized on each worker
• Now, each worker increments cnt locally
• That's it... the cnt value in driver application stays unchanged

As an alternative, you can use Spark's Accumulators to achieve these types of "counters".

Answer 2

In the Spark framework, when you use an external variable in a closure, it is sent automatically to the worker nodes. Each tasks get a new copy of the variable but if you update the variable in the task (this is what happens with your code), the framework does not send it back and synchronize it with the rest of the program, because it would be too expensive to do so.

If you're using external variables in the closures, you can think of it as read-only variables.

If you're trying to count how many elements you mapped, you can use oldRDD.count() / newRDD.count() in the first place (since it seems you don't filter elements, it should give the same result).

Answer 3

我认为您应该先检查一下以了解Spark中的共享变量： http : //spark.apache.org/docs/latest/programming-guide.html#shared-variables