JUnit repeat annotation does not start test case

Question

I have a @Repeat annotation that I use to repeatedly run JUnit tests. The code was taken from this blog post here and modified to run with JUnit 4.10.

Repeat.java

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD})
public @interface Repeat {
    int value();
}

ExtendedRunner.java

public class ExtendedRunner extends BlockJUnit4ClassRunner {

    public ExtendedRunner(Class<?> klass) throws InitializationError {
        super(klass);
    }

    @Override
    protected Description describeChild(FrameworkMethod method) {
        if (method.getAnnotation(Repeat.class) != null &&
                method.getAnnotation(Ignore.class) == null) {
            return describeRepeatTest(method);
        }
        return super.describeChild(method);
    }

    private Description describeRepeatTest(FrameworkMethod method) {
        int times = method.getAnnotation(Repeat.class).value();

        Description description = Description.createSuiteDescription(
                testName(method) + " [" + times + " times]",
                method.getAnnotations());

        for (int i = 1; i <= times; i++) {
            Description d = Description.createSuiteDescription("[" + i + "] " + testName(method));
            d.addChild(Description.createTestDescription(getTestClass().getJavaClass(), testName(method)));
            description.addChild(d);
        }
        return description;
    }

    @Override
    protected void runChild(final FrameworkMethod method, RunNotifier notifier) {

        if (method.getAnnotation(Repeat.class) != null) {

            if (method.getAnnotation(Ignore.class) == null) {
                Description description = describeRepeatTest(method);
                runRepeatedly(methodBlock(method), description, notifier);
            }
            return;
        }

        super.runChild(method, notifier);
    }

    private void runRepeatedly(Statement statement, Description description,
                               RunNotifier notifier) {
        for (Description desc : description.getChildren()) {
            runLeaf(statement, desc, notifier);
        }
    }

}

And finally, the test to run wat() 10 times:

@RunWith(ExtendedRunner.class)
public class RepeatTest {

    private int counter = 0;

    @Repeat(10)
    @Test
    public void wat() {
        System.out.println(counter++);
    }
}

When I run the test, I get the following in the event log.

Failed to start: 9 passed, 1 not started

And this as well:

Interestingly, the test printed from 0 - 9, which makes me believe the tests actually ran 10 times. Yet, I get the above event log. Why does this happen?

Answer 1

I have posted a @Rule on GitHub that repeats tests. This solution is more flexible than using a Runner since it does not preclude the use of another Runner .

RepeatTest Rule

RepeatFailingTest Rule

@ThreadSafe
@ParametersAreNonnullByDefault
public class RepeatTest implements TestRule {

public static RepeatTest findRepeats() {
    return new RepeatTest();
}

private RepeatTest() {
    super();
}

@Override
public Statement apply(final Statement statement, Description description) {
    return new StatementRepeater(statement, description);
}

private final class StatementRepeater extends Statement {

    private final Statement statement;
    private final int repeatCount;

    private StatementRepeater(Statement statement, Description description) {
        super();
        this.statement = statement;
        Repeat repeat = description.getAnnotation(Repeat.class);
        this.repeatCount = getRepeatCount(repeat);
    }

    private final int getRepeatCount(@Nullable Repeat repeat) {
        int count = 1;
        if (repeat != null) {
            count = repeat.count();
        }

        assertThat("Repeat count must be > 0", count,
                OrderingComparison.greaterThan(0));
        return count;
    }

    @Override
    public void evaluate() throws Throwable {
        for (int i = 0; i < repeatCount; i++) {
            statement.evaluate();
        }
    }
}
}