Negative Active Tasks in Spark UI under load (Max number of executor failed)

Question

I am running a spark streaming application on Spark 1.5.0 in CDH 5.5.0. In the logs I see max number of executor failed. I am unable to find the root cause.

We are getting this issue intermittently every other day.Final app status: FAILED, exitCode: 11, (reason: Max number of executor failures reached)

Answer 1

It's a bug, you can track changes in following tickets:

• https://issues.apache.org/jira/browse/SPARK-5098 ,
• https://issues.apache.org/jira/browse/SPARK-10141 ,
• https://issues.apache.org/jira/browse/SPARK-2319

Edit: about this message "Max number of executors failed" - Spark have parameter spark.yarn.max.executor.failures . By default 2x number of executors, minimum 3. If there were more failures than it was set in this parameter, then application will be killed.

You can change value of this parameter. However I would be worried why you have so many executor failures - maybe you've got too less memory? Or bug in code? Without code and/or context information we are not able to help in investigation about potential bug