Failure recovery in spark running on HDinsight

Question

I was trying to get Apache spark run on Azure HDinsight by following the steps from http://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-spark-install/

I was wondering if I have to manage the master/slave failure recovery myself, or will HDinsight take care of it.

Answer 1

I'm also working on Spark Streaming applications on Azure HDInsight. Inside the Spark job, Spark and Yarn can provide some Fault-Tolerance for Master and Slave.

1. But sometimes, the driver and worker will also crash by the user-code error, spark internal issues, and Azure HDInsight issues. So, we need to make our own monitoring/daemon process , and maintain the recovery .
2. For Streaming Scenarios, it's even harder. As the Spark Streaming Job which need keep 7*24 running, there's the concern that how to keep the job recovery for the machine reboot and reimage .