[英]Non-deterministic behavior of skipped tests in parallel testing on Maven Surefire
I have a Maven project with the following expected results when tests are executed ( mvn test
): 我有一个Maven项目,在执行mvn test
时具有以下预期结果( mvn test
):
Tests run: 543, Failures: 0, Errors: 0, Skipped: 8
I was exploring the benefit of running these tests in parallel to see if I could reduce the total elapsed time of the building process. 我正在探索并行运行这些测试的好处,看看我是否可以减少构建过程的总耗用时间。
After adding the following setting to Surefire: 将以下设置添加到Surefire后:
<parallel>methods</parallel>
<threadCount>5</threadCount>
<perCoreThreadCount>true</perCoreThreadCount>
<forkCount>3C</forkCount>
<reuseFork>true</reuseFork>
the number of Skipped tests vary. Skipped测试的数量各不相同。 See the summary below for 50 executions: 有关50次执行,请参阅以下摘要:
01. Tests run: 543, Failures: 2, Errors: 176, Skipped: 8
... 34 runs with similar outputs...
35. Tests run: 543, Failures: 6, Errors: 173, Skipped: 8
36. **Tests run: 543, Failures: 2, Errors: 182, Skipped: 11**
37. **Tests run: 543, Failures: 2, Errors: 176, Skipped: 12**
38. Tests run: 543, Failures: 4, Errors: 177, Skipped: 8
39. Tests run: 543, Failures: 3, Errors: 175, Skipped: 8
40. Tests run: 543, Failures: 4, Errors: 174, Skipped: 8
41. Tests run: 543, Failures: 3, Errors: 172, Skipped: 8
42. Tests run: 543, Failures: 3, Errors: 172, Skipped: 8
43. **Tests run: 543, Failures: 2, Errors: 176, Skipped: 12**
44. Tests run: 543, Failures: 3, Errors: 176, Skipped: 8
... 5 runs with similar outputs...
50. Tests run: 543, Failures: 3, Errors: 172, Skipped: 8
Although it happened in 3/50 of the cases, I was expecting to see different numbers only in Errors
and Failures
but never in Skipped
. 虽然它发生在3/50的案例中,但我期望只在Errors
和Failures
看到不同的数字,但从未在Skipped
看到Skipped
。
Any thoughts about the subject? 关于这个问题的任何想法? Thanks in advance. 提前致谢。
EDIT: 编辑:
FINDING: 发现:
Tests can be ignored in runtime execution: 在运行时执行中可以忽略测试:
Indeed, there are tests with Assumptions in this particular test suite: 实际上,在这个特定的测试套件中有假设的测试:
Assume.assumeTrue(InetAddress.getAllByName(host).length > 1);
Assume.assumeNoException(x);
Assume.assumeTrue(sslContextFactory == null);
Assume.assumeNoException(x);
Assume.assumeTrue(OS.IS_LINUX);
Assume.assumeTrue(OS.IS_LINUX);
Assume.assumeTrue(!OS.IS_WINDOWS);
Assume.assumeTrue(!OS.IS_WINDOWS);
Assume.assumeThat(Integer.parseInt(nano), Matchers.greaterThan(21));
Assume.assumeTrue(false);
Assume.assumeTrue(true);
Assume.assumeTrue(false);
Given the circumstances, I believe the non-determinism is due to Assumptions that can be violated when running tests in parallel. 鉴于这种情况,我认为非确定性是由于在并行运行测试时可能违反的假设。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.