org.apache.spark.SparkException: Task not serializable for date function

Question

I implemented Kryo serialization in spark 1.6.2 in scala. I got "org.apache.spark.SparkException: Task not serializable" for below getdate method.

I am using below getdate function in map transformation to get date.

def getdate(s: String,format_ts:String): Option[Date] = s match {
case "" => null
case _ =>
  val format = new SimpleDateFormat(format_ts)

  Try(new Date(format.parse(s).getTime)).toOption


 }

But when I insert null inplace of date column, it is working fine.

Can someone help me what are the classes need to be part of kryo serialization?.Thanks

Answer 1

I'm not much familiar with Kryo serialization in spark. But according to my experiance SimpleDateFormat is not serializable with spark. SimpleDateFormat class Implemented Serializable in java. But it is not serializable when using with spark( When it parse to the Executor program of hadoop, the class can't be serialized.) I got the same error. You can use LocalDate instead of using SimpleDateFormat for your task. For me LocalDate worked fine in many cases.