Why are some tests not executed in my Gradle build?

Question

I have inherited a project with about 1000 tests. Until recently, all of them have been executed when I entered the gradle clean test command.

From one day to another some of the tests stopped being included in the test statistic (the number of passed, failed and ignored tests that Gradle and Idea output at the end of execution of all tests). The code is still there and the tests were not ignored. Nor was any of the build scripts modified. When I run the tests in IntelliJ Idea they are executed (ie there are no compiler errors that could have explained why the tests don't run).

I believe that the tests are executed, but they results are not included in the overall statistics for the following reason. I executed the command gradle --debug --rerun-tasks clean test > 2017_11_24_gradle.log . Then I looked for the occurrences of one of the missing tests, com.mycompany.comm.CommApplicationTests in the resulting 2017_11_24_gradle.log file (grep "com.mycompany.comm.CommApplicationTests" 2017_11_24_gradle.log).

This is grep's output:

17:00:54.861 [QUIET] [system.out] 17:00:54.860 [DEBUG] [org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor] Executing test class com.mycompany.comm.CommApplicationTests
17:00:54.867 [DEBUG] [TestEventLogger] com.mycompany.comm.CommApplicationTests STARTED
17:00:55.169 [DEBUG] [TestEventLogger] com.mycompany.comm.CommApplicationTests > validateProtocol STARTED
17:00:55.875 [DEBUG] [TestEventLogger] com.mycompany.comm.CommApplicationTests > validateProtocol STANDARD_OUT
17:00:57.272 [DEBUG] [TestEventLogger] com.mycompany.comm.CommApplicationTests > validateProtocol PASSED
17:00:57.273 [DEBUG] [TestEventLogger] com.mycompany.comm.CommApplicationTests > contextLoads STARTED
17:00:57.273 [DEBUG] [TestEventLogger] com.mycompany.comm.CommApplicationTests > contextLoads PASSED
17:00:57.274 [DEBUG] [TestEventLogger] com.mycompany.comm.CommApplicationTests PASSED

From this I conclude that the tests are executed.

The only suspicious thing which may or may not account for the non-inclusion of some test results is this:

16:37:47.251 [ERROR] [system.err] *** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message transform method call failed at JPLISAgent.c line: 844
16:37:47.343 [ERROR] [system.err] *** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message transform method call failed at JPLISAgent.c line: 844
16:37:47.343 [ERROR] [system.err] *** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message transform method call failed at JPLISAgent.c line: 844
16:37:47.346 [ERROR] [system.err] *** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message transform method call failed at JPLISAgent.c line: 844
16:37:47.346 [ERROR] [system.err] *** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message transform method call failed at JPLISAgent.c line: 844
16:37:47.347 [ERROR] [system.err] *** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message transform method call failed at JPLISAgent.c line: 844

These messages are output to System.err during the execution of gradle --debug --rerun-tasks clean test .

How can I fix or work around the erorr (ie make sure that all executed tests are reflected in the end statistics (number of passed, failed, and ignored tests) that Gradle prints at the end)?

Update 1 (27.11.2017 10:49 MSK):

It looks like the error occurs in the code fragment below (comment // Line 844 ):

if ( !errorOutstanding ) {
    jplis_assert(agent->mInstrumentationImpl != NULL);
    jplis_assert(agent->mTransform != NULL);
    transformedBufferObject = (*jnienv)->CallObjectMethod(
                                        jnienv,
                                        agent->mInstrumentationImpl,
                                        agent->mTransform,
                                        loaderObject,
                                        classNameStringObject,
                                        classBeingRedefined,
                                        protectionDomain,
                                        classFileBufferObject,
                                        is_retransformer);
    errorOutstanding = checkForAndClearThrowable(jnienv);
    jplis_assert_msg(!errorOutstanding, "transform method call failed"); // Line 844
}

Update 2 (27.11.2017 16:51 MSK): This error does not occur on Mac (Gradle reports correct numbers of total, passed and failed tests).

Update 3 (29.11.2017 12:48 MSK):

I'm using Java 1.8.0_152 and Gradle 2.14. I cannot upgrade the Gradle version due to the customer requirements.

java -version
java version "1.8.0_152"
Java(TM) SE Runtime Environment (build 1.8.0_152-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)


gradle -version

------------------------------------------------------------
Gradle 2.14
------------------------------------------------------------

Build time:   2016-06-14 07:16:37 UTC
Revision:     cba5fea19f1e0c6a00cc904828a6ec4e11739abc

Groovy:       2.4.4
Ant:          Apache Ant(TM) version 1.9.6 compiled on June 29 2015
JVM:          1.8.0_152 (Oracle Corporation 25.152-b16)
OS:           Windows 10 10.0 amd64

Answer 1

It is hard to tell without seeing the test code. However, what exactly makes you think that the "results are not included in the overall statistics"? I do not see any notion of that? Did the reported number of tests decrease?

Or is it just the error messages? These seem to be "only" error messages reported to the standard error output. Such errors do not necessarily mean the test failed, they are just "warnings" - in the sense nothing is done except for writing them to the error output, which can be seen for instance here .

So, my impression is that the tests indeed passed, they just printed a (warning) error message. The fact they behave slightly differently on Mac may be caused by many circumstances.