根据另一个RDD的值更新RDD

Question

I want to update an rdd based on the values of another rdd. I've tried these three approaches: 1. use left join 2. use subtract by Key and then union 3. use a map and if conditions inside it

However the three approaches mentioned are so slow.

Here's an example: rdd1 contains an rdd based on distinct userID and productID I have. Example, if I have user Id from 0 to 100, I have product Ids from 0 to 100. I have to initially have a rating of 0s for all of them. rdd1 = [(1,1,0.0),(1,2,0.0),(1,3,0.0),...,(100,100,0.0)]

Then rdd2 contains the ratings of specific userIds and productIds. rdd2 = [(1,1,3.0),(100,100,4.0)]

What I want is to include all userIds and productIds in the matrix for collaborative filtering even though there is no rating corresponding to it. I need to do this in order to use explicit ALS in Spark MLLib. If I'm not going to augment 0, I will be getting nonsensical results as the code for explicit does not include scenario wherein there are unobserved values. Thus, they are considered missing instead of zero.

In short, I want to produce this rdd: rdd = [(1,1,3.0),(1,1,0.0),(1,2,0.0),...,(100,100,4.0)]

Do you have some ideas on the fastest way to do this in terms of running time? I have two rdd with millions of entries to be used in updating.

Answer 1

You can simply do:

val res: RDD[(Integer, Integer)] = 
  rdd1.leftOuterJoin(rdd2)
      .mapValues { case (v, wOpt) => wOpt.getOrElse(v) }