使用占位符语法的Spark Scala过滤器

Question

I have following source file

id,name,year,rating,duration

 1. The Nightmare Before Christmas,1993,3.9,4568
 2. The Mummy,1932,3.5,4388
 3. Orphans of the Storm,1921,3.2,9062
 4. The Object of Beauty,1991,2.8,6150
 5. Night Tide,1963,2.8,5126
 6. One Magic Christmas,1985,3.8,5333

I am trying to filter all rows where year=2012 and the following works.

c.map(_.split(",")).filter(x=>x(2).toInt==2012)

But how can I achieve the same using placeholder syntax ( _ ) ???

I could use placeholder syntax ( _ ) in map function (eg rdd.map((_.split(",")) )

Please advice.

Answer 1

Is this what you are looking for

c.map(_.split(",")).filter(_(2).toInt==2012)

But I suggest you to use Spark-CSV to read a csv file like

val df1 = spark.read.option("inferSchema", true)
              .option("header",true)
              .option("delimiter", ",")
              .csv("data1.csv")

and then you can filter easily as

df1.filter($"year" === "2012")

Hope this helps

Answer 2

You can use placeholder by simply doing the following

c.map(_.split(",")).filter(_(2).toInt==2012).map(_.toSeq).foreach(println)

But I would suggest you to go with case class if you know your data is of fixed length

case class row(id: String,
             name: String,
             year: String,
             rating: String,
             duration: String)

and you can use it as

    c.map(_.split(",", -1)).map(array => row(array(0),array(1),array(2),array(3),array(4))).filter(x => x.year.toInt == 2012).foreach(println)

Or to be on the safe side you can combine Option as

c.map(_.split(",", -1)).map(array => {
  row(Option(array(0)) getOrElse "",
    Option(array(1)) getOrElse "",
    Option(array(2)) getOrElse "",
    Option(array(3)) getOrElse "",
    Option(array(4)) getOrElse "")
  })
  .filter(x => x.year.toInt == 2012)
  .foreach(println)