toSeq.toDS() giving java.lang.NullPointerException

Question

Getting nullpointer exception at w.toSeq.toDS() in the below code.

var w = new ListBuffer[String]()
jList match {
  case Some(x) => {
    for (record <- x) {
      w += mapper.writeValueAsString(record)
    }
  }
  case None => println(s"No data read from file : ${f}")
}

import spark.implicits._
val jsonDataSet = w.toSeq.toDS()

Any idea what's wrong here?

Answer 1

It looks like it's not issue with toSeq.toDS() . The piece code which you shared should work without any error/exception. I have prepared test data as below and able to run successfully. Please find sample code as below. It should be issue with some other piece of code. I would request you to share entire code.

import org.apache.spark.sql.SparkSession
import scala.collection.mutable.ListBuffer

object TestToSeqToDs extends App {

  val spark = SparkSession
    .builder()
    .master("local")
    .appName("Test toSeq.toDS() function")
    .getOrCreate()

  def jList(): Option[Seq[String]] = {
    try {
      Some(Seq(("john"), ("Edward")))
    } catch {
      case e: Exception => None
    }
  }

  var w = new ListBuffer[String]()

  jList match {
    case Some(x) => {
      for (record <- x) {
        w += record
      }
    }
    case None => println("That didn't work.")
  }

  import spark.implicits._

  val jsonDataSet = w.toSeq.toDS()
   //OR
  val jsonDataSet = w.toDS()

  println(jsonDataSet.show())

+------+
| value|
+------+
|  john|
|Edward|
+------+

}

Thanks!