使用JUnit运行端到端测试

Question

We are developing a Java command-line app for which we'd like to apply data-driven tests from many many external (properties) files AND allow some knowledgeable users to add tests without touching the Java codebase. Of course, we want to assure that each time we run the app it starts from a clean state (ie no static class side effects, clean file environment...). Here are some options:

(a) Run all tests in a single method in a single JUnit class, calling the app's main() method:

@Test
public void appTest () {
    <get all properties files>
    <for each property file>
        String[] args = <construct command line from property file>
        MyApp.main (args);
        <test result>
     ...
}

does not work as the whole thing is running in a single JVM.

(b) Run all tests in a single method forking the app:

@Test
public void appTest () {
    <get all properties files>
    <for each property file>
        String[] args = <construct command line from property file>
        <fork app with args>
        <test result>
     ...
}

does give us separate JVMs but JUnit (and Surefire) don't know about the individual tests so reporting is pretty useless.

(c) One test per JUnit class:

public class MyAppTest1 {
private static final String PROP_FILE = "src/test/resources/myapp/myapp1.properties
@Test
public void appTest () {
       String[] args = <construct command line from PROP_FILE>
        MyApp.main (args);
        <test result> 
}

}

This works but it's cumbersome and repetitive and you have to add a class for every test.

(d) JUnit parameterized tests (@RunWith(Parameterized.class) are not useful as they run in the same JVM.

(e) Surefire parallelization is WITHIN a JVM.

We are obviously missing something here, as our testing situation is not unusual! Any suggestions much appreciated.

Answer 1

At least a partial answer: Surefire supports forking every test case. Try the following in your surefire configuration section:

<reuseForks>false<reuseForks>
<forkCount>1<forkCount>

Read more about that on http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html

For the general problem with "data driven tests", best would be a custom Junit runner. This is not as hard as you might think, just take a look at the runners package in the Junit sourcecode https://github.com/junit-team/junit/tree/master/src/main/java/org/junit/runners and reimplement/extend the Parameterized Runner using external data.