如何避免surefire测试并行运行

Question

I have an application here where the unit tests are written in a way that they cannot be run in parallel.

When running the tests with maven some of them fail for that reason. I could verify that they are run in parallel by doing a

System.out.println(System.currentTimeMillis() +">>> executing testXXX");
System.out.println(System.currentTimeMillis() +">>> finished  testXXX");

at the start and end of each method. The output is:

1530602546964>>> executing testInstantiation                                                                                                                         
1530602547036<<< finished  testInstantiation                                                                                                                         
1530602547042>>> executing testSimilarNamedResources                                                                                                                 
1530602547050>>> executing testTranslateWithMissingKey                                                                                                               
1530602547051>>> executing testTryTranslateWithMissingKey                                                                                                            
1530602547051<<< finished  testTryTranslateWithMissingKey                                                                                                            
1530602547051>>> executing testTranslationMapWithMissingKey                                                                                                          
1530602547055>>> executing testSilentlyIgnoringExceptionTranslationMapWithMissingKey                                                                                 
1530602547055<<< finished  testSilentlyIgnoringExceptionTranslationMapWithMissingKey   

As we can see after testSimilarNamedResources started, some other tests are started, too.

I tried to configure the surefire plugin to not run in parallel:

<build>                                                                                                                                                           
   <plugins>
     <plugin>
       <groupId>org.apache.maven.plugins</groupId>
       <artifactId>maven-surefire-plugin</artifactId>
       <version>2.22.0</version>
       <configuration>
         <!--parallel>false</parallel-->
         <threadCount>1</threadCount>
         <perCoreThreadCount>false</perCoreThreadCount>
       </configuration>
     </plugin>
   </plugins>
</build>

But it still executes these tests in paralled. I ran mvn with the -X option to see whether my configuration was applied and got this output:

$ mvn -X test | grep -iE "(parallel|threadcount)"                                                                                                                
  <parallel>${parallel}</parallel>                                                                                                                               
  <parallelMavenExecution default-value="${session.parallel}"/>                                                                                                  
  <parallelOptimized default-value="true">${parallelOptimized}</parallelOptimized>                                                                               
  <parallelTestsTimeoutForcedInSeconds>${surefire.parallel.forcedTimeout}<parallelTestsTimeoutForcedInSeconds>                                                   
  <parallelTestsTimeoutInSeconds>${surefire.parallel.timeout}<parallelTestsTimeoutInSeconds>                                                                     
  <perCoreThreadCount default-value="true">false</perCoreThreadCount>                                                                                            
  <threadCount>0</threadCount>                                                                                                                                   
  <threadCountClasses default-value="0">${threadCountClasses}</threadCountClasses>                                                                               
  <threadCountMethods default-value="0">${threadCountMethods}</threadCountMethods>                                                                               
  <threadCountSuites default-value="0">${threadCountSuites}</threadCountSuites>                                                                                  
[DEBUG]   (f) parallelMavenExecution = false                                                                                                                     
[DEBUG]   (s) parallelOptimized = true                                                                                                                           
[DEBUG]   (s) perCoreThreadCount = false                                                                                                                         
[DEBUG]   (s) threadCount = 0                                                                                                                                    
[DEBUG]   (s) threadCountClasses = 0                                                                                                                             
[DEBUG]   (s) threadCountMethods = 0                                                                                                                             
[DEBUG]   (s) threadCountSuites = 0                    

Do I miss something in the plugin configuration?

UPDATE:

I have given up. The behaviour was too strange. Trying to create simple sample didn't work. Those tests were not run in parallel. I didn't found out why this was the case here. We will revamp the whole code and therefore also the unit tests. Not need to find a solution anymore, but it still puzzles me why it showed this strange behaviour…

Answer 1

I would recomend you follow this guidance:

Since of maven-surefire-plugin:2.18, you can apply the JCIP annotation @net.jcip.annotations.NotThreadSafe on the Java class of JUnit test (pure test class, Suite, Parameterized, etc.) in order to execute it in single Thread instance. The Thread has name maven-surefire-plugin@NotThreadSafe and it is executed at the end of the test run. https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html#Parallel_Test_Execution_and_Single_Thread_Execution

Like others have said they should really be rewritten and by exception the tests that absolutely cannot run in parallel should be annotated.

For your plugin definition i suggest you do let it run in parallel as it can be quite a bit faster on modern machines.

I have mine defined as:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>${maven-surefire-plugin.version}</version>
    <configuration>
        <useUnlimitedThreads>true</useUnlimitedThreads>
        <runOrder>random</runOrder>
        <redirectTestOutputToFile>true</redirectTestOutputToFile>
        <rerunFailingTestsCount>1</rerunFailingTestsCount>
        <parallel>methods</parallel>
        <forkedProcessExitTimeoutInSeconds>2</forkedProcessExitTimeoutInSeconds>
    </configuration>
</plugin>

Answer 2

I just spent a few hours dealing with this and ended up resolving it by explicitly setting forkCount to 1. The default is supposed to be 1, but it sure doesn't act like it.

I confirmed this by printing unique names for the running JVM (using ManagementFactory.getRuntimeMXBean().getName()) in a few test cases, each in their own class, along with 30 second Thread.sleep() calls. Without explicitly setting forkCount, sleep() did not block execution of other test cases - teach test clearly ran simultaneously, and getName() returned clearly different values. When I explicitly set forkCount to 1, the sleep() calls blocked execution as expected and getName() returned the same value every time.

Also, just wanted to add @NotThreadSafe is apparently not working as intended either. I tried it first with no luck.