Null值Spark中的左外部联接

Question

I have two RDD and got the left join

left join p1.leftOuterJoin(p2) the result is like:

Array[((String, String), (Int, Option[Int]))] = 
Array(((1001-150329-002-0-04624,5060567),(1,None)), ((1002-141105-008-0-01934,10145500),(1,None)), ((1013-150324-009-0-02270,15750046),(1,None)), ((1005-150814-005-0-05885,5060656),(1,Some(1))), ((1009-150318-004-0-02537,5060583),(1,None)))

I want to replace all None with 0 and get a clean data set like:

Array(((1001-150329-002-0-04624,5060567),0), ((1002-141105-008-0-01934,10145500),0), ((1013-150324-009-0-02270,15750046),0), ((1005-150814-005-0-05885,5060656),1)), ((1009-150318-004-0-02537,5060583),0))

Basically replace all (1,None) with 0 and (1,Some(1)) with 1

Answer 1

If you are looking for the value of the Option when Some or 0 when None , I would implement it with .map and .getOrElse :

a.map { case (k, (_, o)) => (k, o.getOrElse(0)) }

The result match the expected one:

Array(
  ((1001-150329-002-0-04624,5060567),0),
  ((1002-141105-008-0-01934,10145500),0),
  ((1013-150324-009-0-02270,15750046),0),
  ((1005-150814-005-0-05885,5060656),1)),
  ((1009-150318-004-0-02537,5060583),0))