Should JUnit ParentRunner catch AssertionError?

Question

I have implemented a custom TestRunner based on BlockJUnit4ClassRunner .

My assumption was that any failed assertions (indicating product/requirement issues) would be reported to the notifier via addFailedAssumption() while other exceptions would be reported via addFailure() indicating bugs in the unit test itself.

Looking at the results, addFailedAssumption() was never called. In the source code of ParentRunner .runLeaf() , I see

try {
    statement.evaluate();
} catch (AssumptionViolatedException e) {
    eachNotifier.addFailedAssumption(e);
} catch (Throwable e) {
    eachNotifier.addFailure(e);
} finally {
    eachNotifier.fireTestFinished();
}

The exceptions I get are all of type java.lang.AssertionError .

Should ParentRunner catch AssertionError or is there a misunderstanding on my side?

Answer 1

Reading more about this topic, this seems to be a language / translation issue on my side because I'm not a native speaker.

Finding the class Assume helped me getting it right (I hope) and I'll explain it with an example:

Usage of the AssumtionViolatedException

Tests can eg run in different environments, let's say different operating systems. Maybe the product behaves or needs to behave slightly different on different operating systems, eg because it can use an API call on a newer OS which does not exist in an older version of the OS. This may result in code like

if(isApiPresent())
    SimpleAPICall();
else
    // Do some crazy stuff here, potentially slower than the API call

The isApiPresent() call will return different results depending on the OS, so you write 2 unit tests and add an assumption about the environment:

@Test
public void isApiPresent_returns_true_on_Win8()
{
    assumeTrue(System.getProperty("os.version").equals("6.2"));
    assertTrue(isApiPresent());
}
@Test
public void isApiPresent_returns_false_on_Win7()
{
    assumeTrue(System.getProperty("os.version").equals("6.1"));
    assertFalse(isApiPresent());
}

If the assumption about the operating system is not given, the test still gets executed due to the @Test annotation, but it should actually be ignored. The assume...() statements take care of that: they throw an AssumptionViolatedException which can be used to ignore the test.

Eclipse marks a test with a violated assumption (try assumeFalse(true); ) with an ignore icon:

Usage of AssertionError

What I wanted to achieve with my custom implementation of a TestRunner is a bit different. I wanted to find out which unit tests fail due to a requirement issue and which tests fail due to other exceptions which could indicate a bug in the unit test itself, reconstructing the icons in Eclipse. Eclipse already distinguishes these two kinds of issues: AssertionError s are marked with blue icons while Exception s are marked with red icons.

For me, this means that I have to implement the decision in fireTestFailure() :

public void fireTestFailure(Failure failure) {
    originalNotifier.fireTestFailure(failure);

    if (failure.getException().getClass() == AssertionError.class) {
        // Requirement issue
    } else {
        // Unit test issue
    }
}