How can I get a return value from ScalaTest indicating test suite failure?

Question

I'm running a ScalaTest ( FlatSpec ) suite programmatically, like so:

new MyAwesomeSpec().execute()

Is there some way I can figure out if all tests passed? Suite#execute() returns Unit here, so does not help. Ideally, I'd like to run the whole suite and then get a return value indicating whether any tests failed; an alternative would be to fail/return immediately on any failed test.

I can probably achieve this by writing a new FlatSpec subclass that overrides the Scalatest Suite#execute() method to return a value, but is there a better way to do what I want here?

Answer 1

org.scalatest.Suite also has run function, which returns the status of a single executed test.

With a few tweaking, we can access the execution results of each test. To run a test, we need to provide a Reporter instance. An ad-hoc empty reporter will be enough in our simple case:

val reporter = new Reporter() {
  override def apply(e: Event) = {}
}

So, let's execute them:

import org.scalatest.events.Event
import org.scalatest.{Args, Reporter}

val testSuite = new MyAwesomeSpec()
val testNames = testSuite.testNames

testNames.foreach(test => {
  val result = testSuite.run(Some(test), Args(reporter))
  val status = if (result.succeeds()) "OK" else "FAILURE!"
  println(s"Test: '$test'\n\tStatus=$status")
})

This will produce output similar to following:

Test: 'This test should pass'
    Status=OK
Test: 'Another test should fail'
    Status=FAILURE!

Having access to each test case name and its respective execution result, you should have enough data to achieve your goal.