如何将带有double数组的文件转换为spark中的dataframe？

Question

I am new to Scala as well as Apache Spark. My text file contains entries like:

[-0.9704851405656525,1.0286638765434661]
[-0.9704851405656525,1.0286638765434661]
[-1.0353873234576638,-0.001849782262230898]
[-0.9704851405656525,1.0286638765434661]
[-0.9704851405656525,1.0286638765434661]
....

I want to create dataframes from this. To use sql query, My code looks something like this,

def processr(str:String) = str.replaceAll("\\[", "").replaceAll("\\]","")
case class Result(a:Double, b:Double)
val filemat = sc.textFile("mat.txt")
val result = filemat.map(s => s.split(',').map(r=>Result(processr(r[0]).toDouble, processr(r[1]).toDouble)).toDF.cache

And I get error like,

<console>:1: error: identifier expected but integer literal found.
       val result = filemat.map(s => s.split(',').map(r=>Result(processr(r[0]).toDouble, processr(r[1]).toDouble)).toDF.cache

I am not sure, what error I am making in my code. It seems, my split method is not correct. Could anyone suggest me way to covernt into Dataframes? Thanks in advance.

Answer 1

You should use round brackets not the square ones. Extraction from an array in Scala is simply an apply method call:

scala> val r = "[-0.9704851405656525,1.0286638765434661]".split(",")
r: Array[String] = Array([-0.9704851405656525, 1.0286638765434661])

scala> r.apply(0)
res4: String = [-0.9704851405656525

and with some syntactic sugar:

scala> r(0)
res5: String = [-0.9704851405656525

Next your map looks wrong. When you call s.split you get an Array[String] so r is actually a String and r(0) gives you either - or the first digit. You probably want something like this:

filemat.map(_.split(',') match {
  case Array(s1, s2) => Result(processr(s1).toDouble, processr(s2).toDouble)
})

It can be simplified either by using pattern matching with regex:

val p =  "^\\[(-?[0-9]+\\.[0-9]+),(-?[0-9]+\\.[0-9]+)\\]$".r

filemat.map{
   case p(s1, s2) => Result(s1.toDouble, s2.toDouble)
}

or using Row.fromSeq method:

val schema = StructType(Seq(
  StructField("a", DoubleType, false),
  StructField("b", DoubleType, false)))

val p1 = "(-?[0-9]+\\.[0-9]+)".r

sqlContext.createDataFrame(filemat.map(s => 
   Row.fromSeq(p1.findAllMatchIn(s).map(_.matched.toDouble).toSeq)), 
   schema)