Running unit tests on the server (JAX-RS)

Question

I am writing a JAX-RS (Jersey+Maven) application that does some tricky things (eg call native executables embedded in the WAR). I need to run [some of] my unit tests (JUnit4) on the server (Amazon Elastic Beanstalk running Tomcat 7.0.22) to check that everything is ok.

Is there a standard, flexible way of doing this other than RYO (roll your own)? The things I found seem to have more to do with integration testing on the developer machine (ie, Jersey Test Framework). Even RYO is confusing me... how could I call code in the Test Packages from Source Packages?

Basically, I want to create a /test resource that I can call that will return my unit test results from the server in a pretty format. Even better if I could do /test/{category}

Answer 1

I wanted to share what I've learned after posting this question and put up my first answer on StackExchange (a site at which I've arrived countless times through google in search of solutions to my endless problems)

The unit vs integration vs functional testing continuum

There a lot of correcting and arguing and trolling on this subject, so I'd like to clear it up. It's all really very simple. Say you have some service. When you call it there is a chain of events that I'll simplistically illustrate as:

(request received) - (function 1 called) - (function 2 called) - (function 3 called) - (response sent)

Unit testing tests each function (or class or unit) individually in isolation, feeding in an input and checking the output. Integration testing takes several units (such as the function 2-function 3 chain) and also does the ol' in-and-out. Functional testing runs through the entire chain, from request to response. I'll leave it to the reader to guess at some advantages and disadvantages of testing at each level of scale. Anyway, ALL OF THESE TESTS CAN BE RUN IN THE SERVER, AND THERE ARE GOOD REASONS TO RUN THEM THERE.

Types of in-container/in-server testing

• Container-in-the-tests A feature of Spring and other dependency injection frameworks lets you set up a container that's filled only with the bare minimum classes (plus all the mocks) for each of your testss. This is very convenient, since it removes the need for manual wiring and better approximates the production environment. This only allows unit and integration testing.
  • Advantages: a) traditional unit testing (with its advantages of focused and isolated tests) made more convenient b) closer to the production environment since you're testing the autowiring logic e) integrates with IDE test runner f) quick
  • Disadvantages: a) the environment can be rather different from production b) doesn't replace the need for functional testing
• Server-in-the-tests An ordinary test runner runs almost-ordinary unit tests which start up an embedded server or container, and make calls to it. A few frameworks (like Jersey Testing Framework) only allow functional testing, but most (Arquillian, jeeunit) let you do all types. With some of these frameworks, it's as if the tests are running on the server alonside your code and can make any sorts of calls.
  • Advantages (besides the fact you have access to all the container and server services): a) you have self-contained tests and don't need to install or set up anything b) the tests are isolated because a fresh server/container is created for each test or test suite. b) integrates with IDE test runner
  • Disadvantages: a) the environment can be rather different from production (eg, Jetty isn't Tomcat or Glassfish) b) starting/stoping the server slows down the tests c) the frameworks suck. Jeeunit is a tiny project that hasn't even been tested on Windows, Arquillian is big but very new, poorly documented, and I couldn't get it to work either.
• Tests-in-the-server Here, the tests actually are compiled with and run alongside your code.
  • Advantages: a) you have plain, old tests that don't need to be aware of or use any sort of framework
  • Disadvantages: a) no isolation between tests (not necessarily a problem, or even a disadvantage, but may have to take precautions) b) doesn't integrate with IDE test runner (at least in Netbeans)
  • Using Maven during build Maven starts up a server, loads in your special test WAR, executes the tests, and gives a nice Surefire report.
    • Additional advantages: a) it'd done during build (and will integrate with Continuous Integration tools and others) b) no need to install or set up anything (Maven will download, run, etc the server automatically)
    • Additional disadvantages: a) the environment can be rather different (Maven uses Jetty, and it runs on your machine) b) can't re-run in production
  • in-WAR testing Tests are permanently compiled with your code. Whenever and wherever your WAR is up, you can fire up the tests. On your development server, during staging, even in production. This is what my original question was.
    • Additional advantages: a) EXACTLY the correct environment. b) run tests whenever
    • Additional disadvantages: a) need to set up a server

There's one more point to make. Netbeans gives most of the benefits of Maven testing to in-WAR testing. It includes an embedded server, and starts and deploys to it automatically after build. It even open up Firefox... just set it up to point to your /test resource. It's just like doing it the Maven way, but better.

Anyway, I'll show you how to do Maven testing and in-WAR testing together in the same Maven project.

Container-in-the-tests using Spring:

Spring is a sprawling container framework. Its dependency injection mechanisms intertwine with Jax-RS to glorious effect, at the cost of a significant learning curve. I won't explain how Spring or Jax-RS works. I'll jump right into the instructions and hopefully readers can adapt the ideas to other scenarios.

The way to get a container going in your JUnit 4 tests is to use the Spring test runner, declare the classes you'd like to register in the container, register some Jax-RS-specific helper classes, register your mocks, and finally use your Jax-RS resource as if it were an ordinary class:

@RunWith(SpringJUnit4ClassRunner.class)
@WebAppConfiguration
@ContextConfiguration(classes={
    MyClass1.class,
    Myclass2.class,
    MyJaxRsResource.class,
    MockServletContextAwareProcessor.class,
    MyCTest.Config.class
})
public class MyCTest
{
    @Configuration
    static class Config 
    {
          // Set up and register mocks here, and watch them be autowired!
          @Bean public DBService dbJobService() throws DBException
            {
                return mock(DBService.class); 
            }
    }

    @Autowired MyJaxRsResource myResource;

    @Test public void test() {
         String response = myResource.get("hello");
    }
}

@WebAppConfiguration injects its own ServletContextAwareProcessor. However, MockServletContextAwareProcessor is necessary when the path to the unpacked WAR file has to be set dynamically, since WebAppConfiguration only lets you set the path statically at compile time. Using this class when running the-tests-in-the-server (see below), I inject the real ServletContext. I used Spring's profiles feature to suppress it via an environment variable (which isn't very elegant). setServletContext is called simply by the server test runner.

@Configuration
public class MockServletContextAwareProcessor {

public static void setServletContext(ServletContext sc) {
    servletContext = sc;
}    
private static ServletContext getServletContext() {
    return servletContext;
}
private static ServletContext servletContext;    
    
@Configuration
@Profile("server-test")
static class ServerTestContext {

    static public @Bean
    ServletContextAwareProcessor 
        scap() {
            ServletContext sc = getServletContext();
            return new ServletContextAwareProcessor(sc);
    }
}    
}

Server-in-the-tests using Maven:

Step 1) Create regular JUnit tests in the /src/test folder, but name them IT*.java or *IT.java or *ITCase.java (eg, MyClassIT.java) You can name them differently, but this is what Failsafe expects by default. IT stands for integration test, but the test code can lie anywhere on the testing continuum. Eg, you can instantiate a class and unit test it, or you can fire up HttpClient (or Jersey Client), point it at yourself (note the port below), and functionally test your entrypoints.

public class CrossdomainPolicyResourceSTest extends BaseTestClass {

static com.sun.jersey.api.client.Client client;

  @BeforeClass public static void 
startClient() {

        client = Client.create();
    }

  @Test public void 
getPolicy() {

        String response = 
            client
                .resource("http://localhost/crossdomain.xml")
                .get(String.class);

        assertTrue(response.startsWith("<?xml version=\"1.0\"?>"));
    }
}

BaseTestClass is just a little helper class that prints the name of the test class and test as it executes (useful for tests-in-server, see below):

public abstract class BaseTestClass {

@ClassRule public static TestClassName className = new TestClassName();
@Rule public TestName testName = new TestName();    

  @BeforeClass public static void 
printClassName() { 
        System.out.println("--" + className.getClassName() + "--"); 
    }    
  @Before public void 
printMethodName() {
        System.out.print(" " + testName.getMethodName()); 
    }    
  @After public void 
printNewLine() { 
        System.out.println(); 
    }
}

Step 2) Add maven-failsafe-plugin and maven-jetty-plugin to your pom.xml

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>2.11</version>
    <executions>
        <execution>
            <goals>
                <goal>integration-test</goal>
                <goal>verify</goal>
            </goals>
        </execution>
    </executions>
</plugin>
<plugin>
    <groupId>org.mortbay.jetty</groupId>
    <artifactId>maven-jetty-plugin</artifactId>
    <version>6.1.26</version>
    <configuration>
        <!-- By default the artifactId is taken, override it with something simple -->
        <contextPath>/</contextPath>
        <scanIntervalSeconds>2</scanIntervalSeconds>
        <stopKey>foo</stopKey>
        <stopPort>9999</stopPort>
        <connectors>
            <connector implementation="org.mortbay.jetty.nio.SelectChannelConnector">
                <port>9095</port>
                <maxIdleTime>60000</maxIdleTime>
            </connector>
        </connectors>
    </configuration>
    <executions>
        <execution>
            <id>start-jetty</id>
            <phase>pre-integration-test</phase>
            <goals>
                <goal>run</goal>
            </goals>
            <configuration>
                <scanIntervalSeconds>0</scanIntervalSeconds>
                <daemon>true</daemon>
            </configuration>
        </execution>
        <execution>
            <id>stop-jetty</id>
            <phase>post-integration-test</phase>
            <goals>
                <goal>stop</goal>
            </goals>
        </execution>
    </executions>
</plugin>

Step 3) Profit. Really, that's it! Just run 'mvn install' or hit build in the IDE, and the code will build, your regular *Test.java tests will run, the jetty server will start up, the *IT.java tests will run, and you'll get a nice report.

Packaging your tests in your WAR to run anywhere:

(use together or separately from above instructions)

Step 1) Get your test classes (the src/test/ directory) embedded in the WAR by instructing the maven-war-plugin to include them: (adapted from here )

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-war-plugin</artifactId>
    <version>2.1.1</version>
    <configuration>
        <failOnMissingWebXml>false</failOnMissingWebXml>
        <webResources>
            <resource>
                <directory>${project.build.directory}/test-classes</directory>
                <targetPath>WEB-INF/classes</targetPath>
            </resource>
            <resource>
                <directory>${project.build.directory}/test-libs</directory>
                <targetPath>WEB-INF/lib</targetPath>
            </resource>
        </webResources>
    </configuration>
</plugin>

Note: You can create a separate WAR with integrated tests by creating an additional execution and in its configuration set and (the details I leave to the reader)

Note: Ideally, the above would exclude all regular tests (and only copy *IT.java) However, I couldn't get includes/excludes to work.

You will also have to include the test libraries by giving the maven-dependency-plugin an additional execution with a goal of copy-dependency that includes the test scope

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <version>2.1</version>
    <executions>
        <execution>
            <id>copy-dependencies</id>
            <phase>prepare-package</phase>
            <goals>
                <goal>copy-dependencies</goal>
            </goals>
            <configuration>
                <excludeScope>compile</excludeScope>
                <outputDirectory>${project.build.directory}/test-libs</outputDirectory>
                <overWriteReleases>true</overWriteReleases>
                <overWriteSnapshots>true</overWriteSnapshots>
                <overWriteIfNewer>true</overWriteIfNewer>
            </configuration>
        </execution>
    </executions>
</plugin>

If maven-dependency-plugin already has other executions (eg, Netbeans inserts one for javaee-endorsed-api), do not delete them.

Step 2) Programmatically run your tests using JUnitCore (JUnit4).

String runTests() {
    PrintStream sysOut = System.out;
    PrintStream sysErr = System.err;
    ByteArrayOutputStream stream = new ByteArrayOutputStream();
    PrintStream out = new PrintStream(stream);
    try {
        System.setOut(out);
        System.setErr(out);
        TextListener listener = new TextListener(out);
        JUnitCore junit = new JUnitCore();
        junit.addListener(listener);
        
        junit.run(MyClassIT.class,
                  AnotherClassIT.class,
                  ...etc...);

    } finally {
        System.setOut(sysOut);
        System.setErr(sysErr);
        out.close();
    }
    
    return stream.toString();
}

Step 3) Expose your tests via JAX-RS

@Path("/test")
public class TestResource {

    @GET
    @Produces("text/plain")
    public String getTestResults() {
  
        return runTests();
    }

    private String runTests() {
        ...
    }

}

Put this class along with your other test classes (in src/test) so that it could reference them.

However, if you are subclassing the javax.ws.rs.core.Application class where you're registering all your resources, you'll have a problem referencing TestResource (since source code can't reference test code). To work around this, create a completely empty dummy TestResource class under src/main/...[same package]... This trick works because the dummy TestResource will be overwritten by the real one during packaging.

public class ShoppingApplication extends Application {

    @Override
    public Set<Class<?>> getClasses() {
        return new HashSet<Class<?>>() {{
            add(TestResource.class);
        }};
    }

    @Override
    public Set<Object> getSingletons() {
        return new HashSet<Object>();
    }
}

package ...same package as the real TestResource...
public class TestResource {

}

Step 4) Set up your IDE to launch/deploy your app and open your browser point to "/test" automatically after build.

Answer 2

The winning keyword turns out to be "in-container testing". The brand-new and preeminent framework is Arquillian .

Strangely, there doesn't seem to be anything else. Someone else on StackOverflow asked "I don't see any of these projects too widely used, so is there something bad with in-container testing?" But did not receive a clear reply.

I guess it's just a small area between the two large spheres of unit testing and full integration testing that needs to be covered by in-container testing. For me, too, I only need a handful of tests to check if the server resources are accessible and functional. Probably should have wrote them by hand than spent all this time researching (and then learning) in-container testing.

Answer 3

Using Maven, Surefire can give you formatted reports of testing results.

http://maven.apache.org/plugins/maven-surefire-report-plugin/report-mojo.html

There are any number of ways to make the content of those reports available, whether they're sent to you or published to a web page. You have many options.

Answer 4

Jakarta Cactus seems to have done what I am looking for. Its homepage states: "Cactus is a simple test framework for unit testing server-side java code... It uses JUnit... Cactus implements an in-container strategy..." A URL such as http://localhost:8080/test/ServletTestRunner?suite=TestSampleServlet would provide a pretty HTML output.

However, the Apache Foundation put it in the Attic for lack of active development. Does that mean I shouldn't think about using it? The Attic page says "Cactus users are encouraged to switch to other techniques for testing" without explaining what those are!

Answer 5

I don't think there's a standard way, but you could investigate using Spring Remoting to call the methods on the server you're interested in, from your developer machine. If you use interfaces and inject the service you're testing, you should be able to run the same unit test twice, once locally and once on the server, just by changing the Spring config.