在火花壳中使用mapPartitionsWithIndex将rowNumber（）覆盖（partition_index）

Question

I'm trying to add partition index and rownumber in partition to rdd and I did it. But when I tried to get the value of last rownumber I got zero, the rownumber array seemed untouched. Variable scope problem?

It's like rowNumber()/count() over(partition_index) but rownumber added along with partition index in one loop, so maybe more efficient?

Here comes the code:

scala> val rdd1 = sc.makeRDD(100 to 110)
rdd1: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[32] at makeRDD at <console>:25

scala> val rownums=new Array[Int](3)
rownums: Array[Int] = Array(0, 0, 0)

scala> val rdd2=rdd1.repartition(3).mapPartitionsWithIndex( (idx, itr) => itr.map(r => (idx, {rownums(idx)+=1;rownums(idx)}, r)) )
rdd2: org.apache.spark.rdd.RDD[(Int, Int, Int)] = MapPartitionsRDD[37] at mapPartitionsWithIndex at <console>:29

scala> rdd2.collect.foreach(println)
(0,1,100)
(0,2,107)
(0,3,104)
(0,4,105)
(0,5,106)
(0,6,110)
(1,1,102)
(1,2,108)
(1,3,103)
(2,1,101)
(2,2,109)

scala> //uneffected??

scala> rownums.foreach(println)
0
0
0

scala> rownums
res20: Array[Int] = Array(0, 0, 0)

I'm expecting (6,3,2) for rownums:(

---

Solved using Accumulator:

scala> import org.apache.spark.util._
import org.apache.spark.util._

scala> val rownums=new Array[LongAccumulator](3)
rownums: Array[org.apache.spark.util.LongAccumulator] = Array(null, null, null)

scala> for(i <- 0 until rownums.length){rownums(i)=sc.longAccumulator("rownum_"+i)}

scala> val rdd1 = sc.makeRDD(100 to 110)
rdd1: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[92] at makeRDD at <console>:124

scala> val rownums2=new Array[Int](3)
rownums2: Array[Int] = Array(0, 0, 0)

scala> val rdd2=rdd1.repartition(3).mapPartitionsWithIndex( (idx, itr) => itr.map(r => (idx, {rownums2(idx)+=1;rownums(idx).add(1);rownums2(idx)}, r)) )
rdd2: org.apache.spark.rdd.RDD[(Int, Int, Int)] = MapPartitionsRDD[97] at mapPartitionsWithIndex at <console>:130

scala> rdd2.collect.foreach(println)
(0,1,107)                                                                       
(0,2,106)
(0,3,105)
(0,4,110)
(0,5,104)
(0,6,100)
(1,1,102)
(1,2,103)
(1,3,108)
(2,1,109)
(2,2,101)

scala> rownums.foreach(x=>println(x.value))
6
3
2

scala> 

Answer 1

Spark runs in a distributed system. That means you don't have access to modify elements outside your functions.

If you want to get an array with the count of each partition, you need to convert your RDD to an RDD[Int] where each row is the count of of the partition, then collect it.

rdd.mapPartitions(itr => Iterator(itr.size))

If the partition index is important, you can create and RDD[Int,Int] to include it along with the row count.

rdd.mapPartitionsWithIndex((idx, itr) => Iterator((idx, itr.size)))

Answer 2

Please read Understanding closures from the programming guide:

Prior to execution, Spark computes the task's closure. The closure is those variables and methods which must be visible for the executor to perform its computations on the RDD (in this case foreach()). This closure is serialized and sent to each executor.

The variables within the closure sent to each executor are now copies and thus, when counter is referenced within the foreach function, it's no longer the counter on the driver node. There is still a counter in the memory of the driver node but this is no longer visible to the executors! The executors only see the copy from the serialized closure. Thus, the final value of counter will still be zero since all operations on counter were referencing the value within the serialized closure.

You are modifying a local copy of the variable, not the original variable.