I am trying to read a .txt file with |
delimiters as an RDD and trying return a Map[(String, String),(Double, Double)]
, however I am running into CastException
java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Double
input data looks like this
string1|string2|100.00|200.00
string1|string2|34.98|0.989
this is how i am reading the file as rdd and parsing it
val mydata = sc
.textFile("file")
.map(line => line.split("|"))
.map(row =>
((row(0), row(1)),
(row(2).asInstanceOf[Double], row(3).asInstanceOf[Double])))
.collect
.toMap
How can I fix this issue
expected o/p:
Map[(String, String),(Double, Double)] = Map((string1,string2) -> (100.0,200.0), (string1,string2) -> (34.98,0.989))
To be on the safe side you can use trim
function and you can use collectAsMap
val mydata = sc
.textFile("file")
.map(line => line.split("\\|"))
.map(row =>
((row(0), row(1)),
(row(2).trim.asInstanceOf[Double], row(3).trim.asInstanceOf[Double])))
.collectAsMap()
And to be more safe you can use Try/getOrElse
val mydata = sc
.textFile("file")
.map(line => line.split("\\|"))
.map(row =>
((row(0), row(1)),
(Try(row(2).trim.asInstanceOf[Double]).getOrElse(0.0), Try(row(3).trim.asInstanceOf[Double]).getOrElse(0.0))))
.collectAsMap()
Moreover you can use toDouble
instead of asInstanceOf[Double]
val mydata = sc
.textFile("file")
.map(line => line.split("\\|"))
.map(row =>
((row(0), row(1)),
(Try(row(2).trim.toDouble).getOrElse(0.0), Try(row(3).trim.toDouble).getOrElse(0.0)))
)
.collectAsMap().foreach(println)
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.