RDD数据上的迭代Apache Spark

Question

i have a data looks like below consists of latitude and longitude values

45.25,23.45
22.15,19.35
33.24,12.45
15.67,21.22

I need to construct a matrix based on Euclidean distance between two points

as there are 4 points, we will get and 4x4 matrix

p1p1 p1p2 p1p3 p1p4
p2p1 p2p2 ........
..................
..........p4p3 p4p4

Now the question is how can we perform iterations in Apache Spark java ( as below code which is implemented in java )

int nrows = latit.size();
int ncols = longit.size();
double[][] w = new double[nrows][ncols];
for(int i=0;i<nrows;i++) {
       for(int j=0;j<ncols;j++) {
                temp1 = latit.get(i) - latit.get(j);
                temp2 = longit.get(i) - longit.get(j);
                temp3 = Math.pow(temp1, 2) + Math.pow(temp2, 2);
               w[i][j] = Math.sqrt(temp3);
           }
       }

Please suggest the suitable way to store data into RDD and performing iterations in Java API

Answer 1

In Spark you would translate this into a set of transformations and actions. Given pointsRDD containing the location data, then you can obtain the euclidean distance as:

points.cartesian(points).map{case ((x1, y1),(x2,y2)) => math.sqrt((x2-x1)*(x2-x1)+(y2-y1)*(y2-y1))}