两个HashMap的交集（HashMap <Integer,HashSet<Integer> &gt;）Scala中的RDD for Spark

Question

I am working in Scala for programming in Spark on a Standalone machine (PC having Windows 10). I am a newbie and don't have experience in programming in scala and spark. So I will be very thankful for the help.

Problem:

I have a HashMap, hMap1, whose values are HashSets of Integer entries (HashMap>). I then store its values (ie, many HashSet values) in an RDD. The code is as below

val rdd1 = sc.parallelize(Seq(hMap1.values()))

Now I have another HashMap, hMap2, of same type ie, HashMap>. Its values are also stored in an RDD as

val rdd2 = sc.parallelize(Seq(hMap2.values()))

I want to know how can I intersect the values of hMap1 and hMap2

For example:

Input:

the data in rdd1 = [2, 3], [1, 109], [88, 17]

and data in rdd2 = [2, 3], [1, 109], [5,45]

Output

so the output = [2, 3], [1, 109]

Answer 1

Problem statement

My understanding of your question is the following:

Given two RDDs of type RDD[Set[Integer]] , how can I produce an RDD of their common records.

Sample data

Two RDDs generated by

val rdd1 = sc.parallelize(Seq(Set(2, 3), Set(1, 109), Set(88, 17)))
val rdd2 = sc.parallelize(Seq(Set(2, 3), Set(1, 109), Set(5, 45)))

Possible solution

If my understanding of the problem statement is correct, you could use rdd1.intersection(rdd2) if your RDDs are as I thought. This is what I tried on a spark-shell with Spark 2.2.0:

rdd1.intersection(rdd2).collect

which yielded the output:

Array(Set(2, 3), Set(1, 109))

This works because Spark can compare elements of type Set[Integer] , but note that this is not generalisable to any object Set[MyObject] unless you defined the equality contract of MyObject .