I want to get 2 rdd from Cassandra,then join them.And I want to skip the empty value.
def extractPair(rdd: RDD[CassandraRow]) = {
rdd.map((row: CassandraRow) => {
val name = row.getName("name")
if (name == "")
None //join wrong
else
(name, row.getUUID("object"))
})
}
val rdd1 = extractPair(cassRdd1)
val rdd2 = extractPair(cassRdd2)
val joinRdd = rdd1.join(rdd2) //"None" join wrong
use flatMap can fix this,but i want to know how to use map fix this
def extractPair(rdd: RDD[CassandraRow]) = {
rdd.flatMap((row: CassandraRow) => {
val name = row.getName("name")
if (name == "")
seq()
else
Seq((name, row.getUUID("object")))
})
}
This isn't possible with just a map
. You would need to follow it up with a filter
. But you would still be best to wrap the valid result in a Some
. But, then you would still have it wrapped in a Some as a result...requiring a second map
to unwrap it. So, realistically, your best option is something like this:
def extractPair(rdd: RDD[CassandraRow]) = {
rdd.flatMap((row: CassandraRow) => {
val name = row.getName("name")
if (name == "") None
else Some((name, row.getUUID("object")))
})
}
Option
is implicitly convertable to a flattenable type and conveys your methods message better.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.