Running multiple JUnit tests - Don't repeat if a failure happens once

Question

I am in a project now that is using JUnit as a framework to test engineering data (ref: last question Creating a Java reporting project -- would like to use JUnit and Ant but not sure how )

Since a picture (err a code block) tells a 1,000 words, so let me paste my loop:

JUnitCore junit = new JUnitCore();
RunListener listener = new RunListener();
junit.addListener(listener);

[...]

for (AbstractFault fault : faultLog) {
    theFault = fault;
    Result result = junit.run(GearAndBrakeFaultLogReports.class);

    for (Failure f : result.getFailures()) {
        output.println(log.getName());
        output.println(fault.getName());
        output.println(HelperFunctions.splitCamelCase(f.getDescription()
                .getMethodName()));
        output.println(f.getMessage());
        output.println();
    }
}

As you can see, I am running the "junit.run" many times (for each fault in the log).

However, if any one of my tests fires a fail() I don't want to repeat that test. In other words, if there are 50 faults in a log, and in fault #1 a test fails, I don't want to attempt that test in the 49 future faults I am looping through.

Here is an example test:

private static boolean LeftMLGDownTooLongFound = false;
@Test
public final void testLeftMLGDownTooLong() {
if (!LeftMLGDownTooLongFound 
        && handleLDGReportFaults(false)
        && theFault.getName().equals(FaultNames.LDG_LG_DWN_TIME.toString())) {

        assertNotGreater(getPCandRecNum(), 8f, ldgFault.getLeftStrutUpTime());
        LeftMLGDownTooLongFound = true;
    }
}

Currently, do to this, I am making a static bool that is set to false at first, but switches to true after the first assertion. Not sure if this works, but its the idea. I don't want to do this for every single test (100's of them).

Is there any public function, method, or way in the JUnitCore or Runner class that I can flag it so a test never runs more than once after a fail() is called?

Answer 1

Ah, figured it out. To do this, I need to implement a way to find the failed tests, then in the @Before area, ax out of the test. Here is what I added.

@Rule public TestName name = new TestName();
@Before
public void testNonFailedOnly() {
    Assume.assumeTrue(!failedTests.contains(name.getMethodName()));
}
private static List<String> failedTests = new ArrayList<String>(256);

@Rule
public TestWatcher watchman = new TestWatcher() {
    /* (non-Javadoc)
     * @see org.junit.rules.TestWatcher#failed(java.lang.Throwable, org.junit.runner.Description)
     */
    @Override
    protected void failed(Throwable e, Description description) {
        super.failed(e, description);
        failedTests.add(description.getMethodName());
    }
};

It does add about 1.5 seconds of overhead, which sucks... but better than the alternative!!

Anyone have ideas on how to optimize this? I believe the overhead is from the TestWatcher, don't think it from the arraylist.

Answer 2

I used a Java Class that every test extends . In the @Before of this class I set a boolean hasPassed = false; At the end of every @Test method I set this variable hasPassed = true; In the @AfterMethod you can then check the variable. If your test causes an exception, it wont reach the end and the variable is still false.