In spark streaming job, how to collect error messages from executors to drivers and log these at the end of each streaming batch?

Question

I want to log all the error messages in the driver machine. How to do this efficiently.

Answer 1

Basically you need a Custom Logger to write all your application specific logs to separate file instead of writing in YARN logs.

https://mapr.com/blog/how-log-apache-spark/

The above URL says clearly , how you can implement Custom logger mechanism in Apache Spark.

Answer 2

You can put a try/catch block around your code and print the stack-trace.

 try {

    // your code

  } catch {
    case e: Exception => {
      e.printStackTrace()
    }

Answer 3

Depending on the amount of messages to be collected, you could consider doing this using an Accumulator . The built-in ListAccumulator seems to be a prime candidate.