Apache Spark：更新worker中的全局变量

Question

I am curious if the following simple code will work in distributed environment (it does work properly in standalone)?

public class TestClass {
    private static double[][] testArray = new double[4][];
    public static void main(String[] args) {
        for(int i = 0; i<4; i++)
        {
            testArray[i] = new double[10];
        }
        ...
        JavaRDD<String> testRDD = sc.textFile("testfile", 4).mapPartitionsWithIndex(
            new Function2<Integer, Iterator<String>, Iterator<String> >() {
                @Override
                public Iterator<String> call(Integer ind, Iterator<String> s) {
                    /*Update testArray[ind]*/
                }
            }, true
        );
    ...

If it is supposed to work, I wonder how Spark sends the portion of testArray from worker to master node?

Answer 1

No. It's not supposed to work in a distributed environment.

Variables captured in a closure will be serialized and sent to the workers. The data initially set at the driver will be available to the workers, but any updates at the worker level will be only accessible in the local scope.

On local, the variable is in the same memory space and therefore you see the updates, but that will not scale to a cluster.

You need to convert the computation in terms of RDD operations in order to collect the results.