Spark cannot catch Size exceeds Integer.MAX_VALUE error exception?

Question

I get this Exception while making repartition operation(decreasing partition size).

Caused by: java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE

While trying to catch this exception somehow below try-catch block doesn't work. It didn't catch the exception.

 try{
      someDF.repartition(10)
        .persist(StorageLevel.MEMORY_AND_DISK)
        .write.mode("overwrite").format(format).save(temp_location)
    }
    catch {
      case ex: java.lang.IllegalArgumentException => {
      // Do something else
    }

But if I make exception type more generic it started to catch exception.

 try{
      someDF.repartition(10)
        .persist(StorageLevel.MEMORY_AND_DISK)
        .write.mode("overwrite").format(format).save(temp_location)
    }
    catch {
      case ex: Exception => {
      // Do something else
    }

So what is the reason behind it?

Does spark somehow throw an other exception internally, different than written as error message?

Answer 1

Note "caused by" in the exception message; it means IllegalArgumentException is the cause of the exception and you should look at the stack trace before it for the exception class and message itself. It'll look like

Exception in thread "<thread-name>" <exception-class>: <message>

See also Setting Exception cause in java which gives this example stack trace:

Exception in thread "main" java.lang.RuntimeException: Some other message
    at Exceptions.main(Exceptions.java:4)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.lang.RuntimeException: Some message
    at Exceptions.main(Exceptions.java:3)
    ... 5 more