In a scala spark job, running in yarn, how can I fail the job so that yarn shows a Failed status

Question

I have a simple if statement in my scala spark job code, that if false i want to stop the job and mark it failed. I want the yarn UI to show the spark job with a status of failed, but everything i've done so far has stopped the job, but only shows up as successfully finished on the yarn UI.

    if(someBoolen) {
        //context.clearAllJobs()
        //System.exit(-1)
        //etc, nothing so far, stops the job and show as failed in the yarn UI
    }

Any help would be great.

Answer 1

Throwing an exception (and not catching it) will cause the process to fail.

if(someBoolen) {
    throw new Exception("Job failed");
}