在火花流工作中，如何从执行程序收集错误消息到驱动程序并在每个流批处理结束时记录这些错误消息？

Question

I want to log all the error messages in the driver machine. How to do this efficiently.

Answer 1

Basically you need a Custom Logger to write all your application specific logs to separate file instead of writing in YARN logs.

https://mapr.com/blog/how-log-apache-spark/

The above URL says clearly , how you can implement Custom logger mechanism in Apache Spark.

Answer 2

You can put a try/catch block around your code and print the stack-trace.

 try {

    // your code

  } catch {
    case e: Exception => {
      e.printStackTrace()
    }

Answer 3

Depending on the amount of messages to be collected, you could consider doing this using an Accumulator . The built-in ListAccumulator seems to be a prime candidate.