Run Gradle tests with multiple Java toolchains

Question

I've got a Gradle project which uses a Java version specified with the toolchain API:

val minimumJava = JavaLanguageVersion.of(8)
val maximumJava = JavaLanguageVersion.of(16)
java {
    toolchain {
        languageVersion.set(minimumJava)
        vendor.set(JvmVendorSpec.ADOPTOPENJDK)
    }
}

I would like to be able to compile with the minimum supported Java version, then run the tests with all the JDKs the project supports . I tried the following, but apparently only the original tests get executed, all other tests don't, even though the required JDKs get correctly downloaded and set up:

for (javaVersion in JavaLanguageVersion.of(minimumJava.asInt() + 1)..maximumJava) {
    val base = tasks.test.get()
    val testTask = tasks.register<Test>("testUnderJava${javaVersion.asInt()}") {
        javaLauncher.set(
            javaToolchains.launcherFor {
                languageVersion.set(javaVersion)
            }
        )
        classpath = base.classpath
        testClassesDirs = base.testClassesDirs
        isScanForTestClasses = true
    }
    tasks.test.configure { finalizedBy(testTask) }
}

Here is a run in a dumb terminal:

❯ TERM=dumb ./gradlew test testUnderJava10 --rerun-tasks --scan
Picked up _JAVA_OPTIONS: -Dawt.useSystemAAFontSettings=on

<<<SNIP>>>

> Task :testClasses

> Task :test
Picked up _JAVA_OPTIONS: -Dawt.useSystemAAFontSettings=on

Gradle Test Executor 4 STANDARD_OUT
    ~~~ Kotest Configuration ~~~
    -> Parallelization factor: 1
    -> Concurrent specs: null
    -> Global concurrent tests: 1
    -> Dispatcher affinity: true
    -> Default test timeout: 600000ms
    -> Default test order: Sequential
    -> Default isolation mode: SingleInstance
    -> Global soft assertions: false
    -> Write spec failure file: false
    -> Fail on ignored tests: false
    -> Spec execution order: SpecExecutionOrder
    -> Remove test name whitespace: false
    -> Append tags to test names: false
    -> Extensions
      - io.kotest.engine.extensions.SystemPropertyTagExtension
      - io.kotest.core.extensions.RuntimeTagExtension
      - io.kotest.engine.extensions.RuntimeTagExpressionExtension


org.danilopianini.template.test.Tests > A greeting should get printed STARTED

org.danilopianini.template.test.Tests > A greeting should get printed STANDARD_OUT
    [:hello=SUCCESS]

    > Task :hello
    Hello from Danilo Pianini

    BUILD SUCCESSFUL in 2s
    1 actionable task: 1 executed


org.danilopianini.template.test.Tests > A greeting should get printed PASSED

<<<Other tests have no output!>>>

> Task :testUnderJava9
> Task :testUnderJava8
> Task :testUnderJava16
> Task :testUnderJava15
> Task :testUnderJava14
> Task :testUnderJava13
> Task :testUnderJava12
> Task :testUnderJava11
> Task :testUnderJava10

BUILD SUCCESSFUL in 23s
36 actionable tasks: 36 executed

<<<SNIP>>>

From the build scan , it appears that tests are not executed but those with JDK8. I'm puzzled, the docs say that this should be straightforward:

tasks.register<Test>("testsOn14") {
    javaLauncher.set(javaToolchains.launcherFor {
        languageVersion.set(JavaLanguageVersion.of(14))
    })
}

Answer 1

I think I worked out the root cause of the issues I was experiencing, I'm posting the solution in case someone else runs into similar issues. I had the following tests configuration:

tasks.test {
    useJUnitPlatform()
    testLogging {
        showStandardStreams = true
        showCauses = true
        showStackTraces = true
        events(*org.gradle.api.tasks.testing.logging.TestLogEvent.values())
        exceptionFormat = org.gradle.api.tasks.testing.logging.TestExceptionFormat.FULL
    }
}

Which was instructing the task called test to useJunitPlatform() . This setting does not get automatically propagated to all subsequent Test tasks (of course). So, in this case, the solution is simply to use instead:

tasks.withType<Test> { 
    // Same configuration as above
}

Update 2022-03-16

I decided to create a multi-JVM testing plugin for Gradle , so that all the test tasks get created and much less boilerplate is required across projects.

Update 2023-01-17

Gradle recommends using the task configuration avoidance API .

tasks.withType<Test>().configureEach { 
    // Same configuration as above
}