任务不可序列化异常

Question

For some reason I am getting a Task not serializable exception with the following code. I am running this on spark local using sbt test.

@RunWith(classOf[JUnitRunner])
class NQTest extends FeatureSpec with Matchers with Serializable {
  val conf = new SparkConf().setAppName("NQ Market Makers Test").setMaster("local")
  val sc = new SparkContext(conf)
  ...

  val testData : RDD[(String, String)] = sc.textFile("testcases/NQIntervalsTestData").map { line => (line.split(":", 2)(0), line.split(":", 2)(1)) }
  testData.persist();
  def testDatasets(input : Int) = {
    testData.filter(_ match {
      case (s, _) => (s == "Test Case " + input)
      case _      => false
    }).map(x => x match {
      case (_, line) => line
    })
  }

  ...

  feature("NQIntervals") {
    scenario("Test data sanity check") {
      (testDatasets(1).collect()) should not be null
    }
  }
}

And the exception:

org.apache.spark.SparkException: Task not serializable
        at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:166)
        at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:158)
        at org.apache.spark.SparkContext.clean(SparkContext.scala:1623)
        at org.apache.spark.rdd.RDD.filter(RDD.scala:303)
        at test.scala.org.<redacted>.NQTest$.testDatasets(NQTest.scala:31)

Unlike the other stack overflow questions that I've seen here regarding this exception, this seems to be concerning the RDD itself rather than the function I've passed to filter.

For example, we can remove the filter and map entirely and we still end up an exception during the collect. From my googling I've only been able to find answers to problems involving non serializable objects inside a filter or a map, not problems with the RDD itself.

Things I've tried so far:

• Removed the filter and map inside the testDatasets method and just returned the testData set. This caused the exception to happen when collect was called.
• Removed the unit testing framework entirely, made NQTest extend Serializable directly and wrote a one line main method consisting of testDatasets(1).collect() : still the same exception
• Removed testData.persist() : still the same exception

Any insight would be welcome!

Answer 1

Turns out I was a huge idiot and was stopping the spark context before the actual tests were being run. Disregard