如何保持 RDD 的持久性和一致性？

Question

I have the following code (simplification for a complex situation):

val newRDD = prevRDD.flatMap{a =>
    Array.fill[Int](scala.util.Random.nextInt(10)){scala.util.Random.nextInt(2)})
}.persist()
val a = newRDD.count
val b = newRDD.count

and even that the RDD supposed to be persisted (and therefore consistent), a and b are not identical in most cases.

Is there a way to keep the results of the first action consistent, so when the second "action" will be called, the results of the first action will be returned?

* Edit *

The issue that I have is apparently caused by zipWithIndex method exists in my code - which creates indices higher than the count. I'll ask about it in a different thread. Thanks

Answer 1

There is no way to make sure 100% consistent.

When you call persist it will try to cache all of partitions on memory if it fits. Otherwise, It will recompute partitions which are not fit on memory.