当左侧键为Option []时，Spark左外连接

Question

I have

val rdd1 :  RDD[(Option[String], (String, Option[Context]))]

and

val rdd2 : RDD[(String,Double)]

Now, I would like to rdd1.leftOuterJoin(rdd2) but of course I can't because Option[String] is different than String .

The rationale for the join operation is that in case rdd1's key contains some value, I would like to have an additional info on it. The desired output is of type: RDD[(Options[String],((String, Option[Context]),Option[Double]))

What's the bypass?

Answer 1

You can simply map rdd2 to RDD[(Option[String], Double)] :

rdd1.leftOuterJoin(rdd2.map{case (k, v) => (Option(k), v)})

If Context can be expressed using Spark SQL types then you can simply convert both RDDs to DataFrames and join. None are mapped to NULLs so everything should work as expected.