Maven Surefire插件+并行构建

Question

In our fairly large project (around 600 maven modules) we've decided to try maven's parallel build capability. So I've installed maven 3.3 and tried to run unit tests.

I've figured out that sometimes our unit tests fail. If I run the module 'alone' it works, and of course if we run maven 'sequentially' (no -T option) it also works.

So, I think that its due to the fact that sometimes tests interfere, probably some static code or shared instances cause these failures.

My question is, what is the optimal way to run the unit tests of my project? I'm aware of option to spawn JVM upon each test running, but since we have thousands of unit tests I'm afraid such a build would last forever :)

AFAIK there are 3 possible solutions:

1. Just skip the tests, compile/package/install everything in parallel. And then run tests separately. This approach can be feasible for jenkins, but I can be a hassle for developers who are just accustomed to run mvn install

2. Somehow extend surefire plugin that it would automatically rerun N times the unit test if it fails. If the test fails, say 50% of times - its really unstable and should cause the whole build failure. I wouldn't like to use a @RunWith annotation because of two reasons: a. There are just to many tests to update with this annotation and we don't have a base class or something. b. Some tests already contain this annotation (for power mock for example, or junit rules)

3. Somehow extend surefire plugin to run tests in different classloader. This solution has come to my mind due to the fact that maven seems to run multiple modules with the same classloader (I've just put 2 tests that print the address of classloader in two different submodules and ensured that the classloader is the same, the addresses are identical). This solution looks interesting but relatively complicated.

Before diving to any of these solution and talking to my managers, I would appreciate if someone could comment on one of these solutions/provide another one. I Just don't want to reinvent the wheel and hopefully save some time :)

Thanks a lot in advance

Answer 1

How about running each test class in a separate JVM? (And possibly reuse the forked JVMs in other tests).

Honestly, it looks like you didn't give it a shoot. Check out the doc https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html

Just a copy paste of the example:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.19</version>
  <configuration>
    <forkCount>3</forkCount>
    <reuseForks>true</reuseForks>
    <argLine>-Xmx1024m -XX:MaxPermSize=256m</argLine>
    <systemPropertyVariables>
        <databaseSchema>MY_TEST_SCHEMA_${surefire.forkNumber}</databaseSchema>
    </systemPropertyVariables>
    <workingDirectory>FORK_DIRECTORY_${surefire.forkNumber}</workingDirectory>
  </configuration>
</plugin>

By the way, surefire can re-run failing tests: https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html . I'm just afraid you won't be able to set a threshold.

Also not that forking can make debugging (eg, attaching a debugger) quite difficult.

From my experience, not all the surefire settings are compatible with forking so you may need to go step by step while tuning the configuration to see what is actually working and what is not.