Apache Spark Java 8中的自连接示例

Question

I have a data set as below

Kolkata,30
Delhi,23
Lucknow,33
Lucknow,36
Delhi,31
Kolkata,34
Delhi,21
Kolkata,23

Want to do a self join to get a result set of the form.

Lucknow -> (30, 36), Kolkata -> (30, 34), (34, 23), (23, 30).

How can this be achieved using Spark RDD.

JavaPairRDD<String, Integer> words = file.mapToPair(s ->{
    String[] temp = s.split(",");
    return new Tuple2<String, Integer>(temp[0], new Integer(temp[1]));
});        

JavaPairRDD<String, Iterable<Integer>> temp1 = words.groupByKey();        
JavaPairRDD<String, Iterable<Integer>> temp2 = words.groupByKey();            
JavaPairRDD<String, Tuple2<Iterable<Integer>, Iterable<Integer>>> words3 = temp2.join(temp1);

How to iterate now on the tuple?

Answer 1

To get an RDD of <String, Iterable<Int,Int>> back use .groupByKey() after join.

JavaPairRDD<String, Iterable<Tuple2<Integer>>> result = 
      words.join(words).groupByKey()

It is possible I have the Java type wrong, but this is the right sequence of operations. I am more at ease with Python and Scala where the types for result RDDs don't need to be specified. The Spark calls in these other languages perform the same data operations.

Note: groupByKey() is known for being slow and resource intensive on large datasets

After word.join(words) you should have an RDD with elements like:

 (Kolkata, (30,34))
 (Kolkata, (30,23))
 ...

And .groupByKey() groups together all of the values by key so that there is only one row for each key, like this:

 (Kolkata, [(30,34), (30,23), (34,23), ...])

Answer 2

For every unique key, a rdd self join yields to permutation pairs of values.

To remove the duplicate pairs, (Considering your data has no repeated values for a unique key), the filterduplicates() will remove those duplicate value pair from the set of permutations.

//outside main function
type pair=(String,(Int,Int))

def filterduplicates(p:pair):Boolean={
return p._2._1<p._2._2
}    

//insidemain
val rdd=sc.textFile("../cities.txt")
val mapped=rdd.map(l=>l.split(",")).map(l=>(l(0),l(1).toInt))

val joined=mapped.join(mapped) 
//(Lucknow,(33,33)) (Lucknow,(33,36)) (Lucknow,(36,33)) (Lucknow,(36,36))

val grouped=joined.filter(filterduplicates)
//(Lucknow,(33,36))

val listt=grouped.groupByKey().mapValues(_.toList)
val finalresult=listt.collect()
finalresult.foreach(println)
//(Delhi,List((23,31), (21,23), (21,31)))(Kolkata,List((30,34),(23,30)  (23,34))) (Lucknow,List((33,36)))