在Apache Spark中捕获自定义异常

Question

I have a problem to catch custom exceptions in Apache Spark.

When I am doing a validation on a dataset in a foreach loop like this

ds.foreach(
        entry=> {
          validate(entry)
        })

The validate function throws a Custom Exception when the entry is not valid.

But in the catch block I am not able to catch my custom exception only a SparkException is thrown and can be catched:

case customException : CustomException =>
    //is never catched
case exception : SparkException =>
    //can be catched

How can I deal with that? I need to catch different sort of exceptions which are all thrown by the validate method. One way is to read the message of the SparkException which contains the origin exception but this is probably not a nice design.

Any ideas?

Answer 1

Instead of matching base exception, try to match the cause:

import org.apache.spark.rdd.RDD

def ignoreArithmeticException(rdd: RDD[java.lang.Integer]) = try {
  rdd.foreach(1 / _)
} catch {
  case e: SparkException => e.getCause match  {
    case _: java.lang.ArithmeticException => 
      println("Ignoring ArithmeticException")
    case _ => throw e
  }
}

This would for catch:

Try(ignoreArithmeticException(sc.parallelize(Seq(0))))

00/00/00 00:00:00 ERROR Executor: Exception in task 3.0 in stage 35.0 (TID 143)
java.lang.ArithmeticException: / by zero
    at
    ...
Ignoring ArithmeticException
res42: scala.util.Try[Unit] = Success(())

(although in a pretty verbose way), but wouldn't catch:

Try(ignoreArithmeticException(sc.parallelize(Seq(null))))

00/00/00 00:00:00 ERROR Executor: Exception in task 3.0 in stage 38.0 (TID 155)
java.lang.NullPointerException
    at 
   ...
res52: scala.util.Try[Unit] =
Failure(org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 38.0 failed 1 times, most recent failure: Lost task 3.0 in stage 38.0 (TID 155, localhost, executor driver): java.lang.NullPointerException ....