尝试从RDD返回Map [（String，String），（Double，Double）]时无法将java.lang.String强制转换为java.lang.Double错误

Question

I am trying to read a .txt file with | delimiters as an RDD and trying return a Map[(String, String),(Double, Double)] , however I am running into CastException

java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Double

input data looks like this

string1|string2|100.00|200.00
string1|string2|34.98|0.989

this is how i am reading the file as rdd and parsing it

val mydata = sc
  .textFile("file")
  .map(line => line.split("|"))
  .map(row =>
    ((row(0), row(1)),
     (row(2).asInstanceOf[Double], row(3).asInstanceOf[Double])))
  .collect
  .toMap

How can I fix this issue

expected o/p:

Map[(String, String),(Double, Double)] = Map((string1,string2) -> (100.0,200.0), (string1,string2) -> (34.98,0.989))

Answer 1

To be on the safe side you can use trim function and you can use collectAsMap

val mydata = sc
  .textFile("file")
  .map(line => line.split("\\|"))
  .map(row =>
    ((row(0), row(1)),
      (row(2).trim.asInstanceOf[Double], row(3).trim.asInstanceOf[Double])))
  .collectAsMap()

And to be more safe you can use Try/getOrElse

val mydata = sc
  .textFile("file")
  .map(line => line.split("\\|"))
  .map(row =>
    ((row(0), row(1)),
      (Try(row(2).trim.asInstanceOf[Double]).getOrElse(0.0), Try(row(3).trim.asInstanceOf[Double]).getOrElse(0.0))))
  .collectAsMap()

Moreover you can use toDouble instead of asInstanceOf[Double]

val mydata = sc
  .textFile("file")
  .map(line => line.split("\\|"))
  .map(row =>
    ((row(0), row(1)), 
      (Try(row(2).trim.toDouble).getOrElse(0.0), Try(row(3).trim.toDouble).getOrElse(0.0)))
  )
  .collectAsMap().foreach(println)