Getting list of tests from JUnit command line

Question

My problem, in short, is that I want to run some JUnit tests from a command line and I would like to be able to see:

• A list of all tests (whether they ran or not).
• A list of all tests which passed.
• A list of all tests which failed

The reason for the first bullet, is because I believe if one test calls System.exit, the others won't run, I still want to know about these tests, even if they didn't run.

Lets say I have this test class:

    import junit.framework.TestCase;
    public class ALoadOfTests extends TestCase {

    public void testFoo() {
        int a = 5;
        int b = 10;
        int result = a + b;
        assertEquals(15, result);
    }

    public void testBar() {
        fail();
    }

    public void testBaz() {
        fail();
    }       
}

If I run this using the commands:

javac -cp ".;C:\Users\username\Desktop\Test\junit-4.11.jar;C:\Users\username\Desktop\Test\hamcrest-core-1.3.jar" ALoadOfTests.java
java -cp ".;C:\Users\username\Desktop\Test\junit-4.11.jar;C:\Users\username\Desktop\Test\hamcrest-core-1.3.jar" junit.textui.TestRunner ALoadOfTsts

The output is:

.F.F.
Time: 0.002
There were 2 failures:
1) testBar(ALoadOfTests)junit.framework.AssertionFailedError
        at ALoadOfTests.testBar(ALoadOfTests.java:12)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
2) testBaz(ALoadOfTests)junit.framework.AssertionFailedError
        at ALoadOfTests.testBaz(ALoadOfTests.java:16)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

FAILURES!!!
Tests run: 3,  Failures: 2,  Errors: 0

The problem here is that the output doesn't tell me the name of testFoo which exists and passed. Furthermore, if I change one of them to use System.exit, then we don't get much output at all, we just get:

.F.F.

Ideally, to be able to get all the information I need, if I could print out a list of test names first, before any tests are ran (and any chance of System.exit) then I can see which passed and which failed (or didn't run).

If I could get output that just looks similar to:

Tests:
testFoo
TestBar
TestBaz
Results:
TestFoo(passed)
TestBar(failed:reason)

Then I could safely assume:

TestFoo passed
TestBar failed
TestBaz never ran, probably because some test called system.exit.

Is this possible using a custom runner? If it is, any hints on where to start would be greatly appreciated.

Unfortunately I have no control over the code, it should be assumed that it may contain System.exit, possibly, in every method. I also have little control over the tests, so printing out the test names in the setup method isn't ideal.

Thanks,

Answer 1

Changing JUnit output

As described in this blog you can write your own test listener and print out your own messages for when tests pass or fail.

public class ExecutionListener extends RunListener
{
... //feel free to override other methods too

/**
 *  Called when an atomic test has finished, whether the test succeeds or fails.
 * */
public void testFinished(Description description) throws java.lang.Exception
{
    System.out.println("Finished execution of test case : "+ description.getMethodName());
}

/**
 *  Called when an atomic test fails.
 * */
public void testFailure(Failure failure) throws java.lang.Exception
{
    System.out.println("Execution of test case failed : "+ failure.getMessage());
}

/**
 *  Called when a test will not be run, generally because a test method is annotated with Ignore.
 * */
public void testIgnored(Description description) throws java.lang.Exception
{
    System.out.println("Execution of test case ignored : "+ description.getMethodName());
}

}

Then write your own class with a main to run Junit (instead of running the junit.jar)

public class ExecuteWithRunListener
{
    public static void main(String[] args)
    {
        JUnitCore runner = new JUnitCore();
        //Adding listener here
        runner.addListener(new ExecutionListener());
        runner.run(TestFeatureOne.class, TestFeatureTwo.class);
    }
} 

How to prevent calls from System.exit from actually exiting

Your tests can use their own SecurityManager to prevent calls to System.exit from doing anything. The code below will make any calls to System.exit() throw a special Exception which we can catch in our tests to make sure we called exit and to also capture what the exit code was:

public static final SecurityManager NON_EXITABLE_MANAGER = new SecurityManager(){

    @Override
    public void checkPermission(Permission perm) {
        //allow everything
    }
    /**
     * Throws a {@link TriedToExitException} instead of exiting.
     * <p/>
     * {@inheritDoc}
     */
    @Override
    public void checkExit(int status) {
        throw new TriedToExitException(status);
    }

};

public static final class TriedToExitException extends SecurityException{
    private static final long serialVersionUID = 1L;
    private final int exitCode;

    public TriedToExitException(int exitCode){
        this.exitCode=exitCode;
    }

    @Override
    public String getMessage() {
        return String.format("tried to System.exit(%d)",exitCode);
    }

    public int getExitCode() {
        return exitCode;
    }

}

Then in your unit tests you replace the SecurityManager with our version that stops calls to exit (and save the old SecurityManager so we can restore it after the test)

private SecurityManager previousManager=null;


@Before
protected void replaceManager() throws Throwable {
    previousManager = System.getSecurityManager();
    System.setSecurityManager(NON_EXITABLE_MANAGER);
}

@After
protected void restoreManager() {
    System.setSecurityManager(previousManager);
}

 @Test
    public void testCallToExit() throws IOException{

         try{
           //something that calls System.exit
            fail("should call exit");
         }catch(TriedToExitException e){
             //make sure we get correct exit code
             assertEquals(-1, e.getExitCode());

         }
    }