How do I sbt test continually until a flakey test fails?

Question

If there is a test that fails only sometimes, can I ask sbt to run tests continually until failure?

Otherwise, I'm stuck hitting up-arrow while watching Arrow. (pun unintended but leavening)

https://stackoverflow.com/q/22366719/1296806

Answer 1

I had the same problem, and I've ended up implementing this as a command.

lazy val root = Project("test", file(".")).
  settings(commands += Command.command("testUntilFailed") { state =>
    "test" :: "testUntilFailed" :: state
  })

The advantage is that it will skip the SBT loading. You can also add extra parameter or run testOnly to test a single test.

Answer 2

老问题，但我自己也有同样的需求，这里有一个解决方案：sbt将返回非零退出代码，如果你用测试运行一次，所以，一种循环直到失败的方法是只看一下shell中的退出代码：

while [ $? -eq 0 ]; do sbt test; done

Answer 3

I have created sbt plugin for flaky test detection: sbt-flaky . You can run test for:

• specified duration sbt clean "flaky duration=30" ,
• specified times sbt clean "flaky times=30"
• until first failure sbt clean "flaky firstFail" .

Advantage of this plugin is aggregation of failures, history trends and possibility of adding flaky test detection into pipeline.

Answer 4

In my opinion it doesn't particularly apply to SBT or any other build management tool. There's no built-in SBT feature for this. You'd have to build one, but it'd be far from what SBT is meant to offer - build configuration management. It can take quite a while to hit the case which causes the build/test fail. It's very unpredictable.

When your test fails, it means that there are cases the test doesn't pass. The error should tell you what they are that you use to improve the test.

ScalaCheck: Property-based testing for Scala might be of some help.