如果选项类型为 None (Scala)，则使用 getOrElse 不返回任何内容

Question

I am using RDD to create a left outer join as so far I have the following results:

scala> LeftJoinedDataset.foreach(println)
(300000004,Trevor,Parr,Some((35  Jedburgh Road,PL23 6BA)))
(300000006,Ava,Coleman,None)
(200000008,Lisa,Knox,None)
(100000007,Dorothy,Thomson,None)
(400000002,Jasmine,Miller,Some((68  High Street,LE16 3PH)))
(300000009,Ruth,Campbell,None)
(100000005,Deirdre,Pullman,Some((63  Crown Street,SW99 2HY)))
(100000010,Dominic,Parr,None)
(100000001,Simon,Walsh,Some((99  Newgate Street,PA5 9UY)))
(100000003,Liam,Brown,Some((9  Earls Avenue,ML12 2DY)))

To remove the None and Some I have so far used the below getOrElse code:

scala> val LeftJoinedDataset = LeftJoin.map(x=>(x._1,x._2._1._1,x._2._1._2,x._2._2.getOrElse(None)))

This prints out:

scala> LeftJoinedDataset.foreach(println)
(300000004,Trevor,Parr,(35  Jedburgh Road,PL23 6BA))
(300000006,Ava,Coleman,None)
(200000008,Lisa,Knox,None)
(100000007,Dorothy,Thomson,None)
(400000002,Jasmine,Miller,(68  High Street,LE16 3PH))
(300000009,Ruth,Campbell,None)
(100000005,Deirdre,Pullman,(63  Crown Street,SW99 2HY))
(100000010,Dominic,Parr,None)
(100000001,Simon,Walsh,(99  Newgate Street,PA5 9UY))
(100000003,Liam,Brown,(9  Earls Avenue,ML12 2DY))

Although the some has gone, I still want to remove the None and return no data. Eg

(300000006,Ava,Coleman) instead of (300000006,Ava,Coleman,None)

How can i do this?

Many Thanks

Answer 1

You can't have different amount of columns in different rows of the same dataset, so you'll have to either drop that column altogether, all deal with Option values, or fill them with something else (eg empty strings). But just having an Option in that column seems like the best way - it will show the consumer, that this data may be absent.