字符串RDD连接操作

Question

I have RDD of Strings in scala. The strings are id's. It would be something like this.

1
2
3
4

I have another RDD with (id, name) like this.

(1, Name1)
(2, Name2)
(3, Name3)
(4, Name4)
(5, Name5)
(6, Name6)

Now I want to the get names for all the ids in the first RDD. How do I do this?

I realized that if the first RDD was a pairRDD I could just join the two RDD's. So why do we only have join operations for pairRDD?

Answer 1

尝试这个：

rdd1.map(x => (x, null)).join(rdd2).mapValues(x => x._2)

Answer 2

Based on your comment to CafeFeeds answer you could consider a 'broadcast join' if the ids RDD is small enough.

val ids: RDD[Int] = ???
val names: RDD[(Int, String)] = ???
val bcIds = sc.broadcast(ids.collect.toSet)
val result = names.filter(x => bcIds.value.contains(x._2))

The benefit of this is that you don't need to shuffle the names RDD so if it is significantly larger you'll reduce the amount of work that needs to be done significantly. Other than that the simple join method is best.