[英]In a scala spark job, running in yarn, how can I fail the job so that yarn shows a Failed status
I have a simple if statement in my scala spark job code, that if false i want to stop the job and mark it failed. 我在我的scala spark作业代码中有一个简单的if语句,如果为false,我想停止该作业并将其标记为失败。 I want the yarn UI to show the spark job with a status of failed, but everything i've done so far has stopped the job, but only shows up as successfully finished on the yarn UI. 我希望纱线UI以失败状态显示Spark作业,但是到目前为止,我所做的一切都停止了该作业,但仅在纱线UI上显示为成功完成。
if(someBoolen) {
//context.clearAllJobs()
//System.exit(-1)
//etc, nothing so far, stops the job and show as failed in the yarn UI
}
Any help would be great. 任何帮助都会很棒。
Throwing an exception (and not catching it) will cause the process to fail. 引发异常(而不是捕获异常)将导致进程失败。
if(someBoolen) {
throw new Exception("Job failed");
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.