如何调查 jenkins 中的 Java 堆空间错误

Question

I have a matrix pipeline job which performs multiple stages (something like ~200) most of which are functional tests whose results are recorded by the following code:

        stage('Report') {
            script {
               def summary = junit allowEmptyResults: true, testResults: "**/artifacts/${product}/test-reports/*.xml"
               def buildURL = "${env.BUILD_URL}"
               def TestAnalyzer = buildURL.replace("/${env.BUILD_NUMBER}", "/test_results_analyzer")
               def TestsURL = buildURL.replace("job/${env.JOB_NAME}/${env.BUILD_NUMBER}", "blue/organizations/jenkins/${env.JOB_NAME}/detail/${env.JOB_NAME}/${env.BUILD_NUMBER}/tests")
               slackSend (
                  color: summary.failCount == 0 ? 'good' : 'warning',
                  message: "*<${TestsURL}|Test Summary>* for *${env.JOB_NAME}* on *${env.HOSTNAME} - ${product}* <${env.BUILD_URL}| #${env.BUILD_NUMBER}> - <${TestAnalyzer}|${summary.totalCount} Tests>, Failed: ${summary.failCount}, Skipped: ${summary.skipCount}, Passed: ${summary.passCount}"
               )
            }
        }

The problem is that this Report stage regularly fails with the following error:

> Archive JUnit-formatted test results                        9m 25s
[2022-11-16T02:51:49.569Z] Recording test results
Java heap space

I have increased the heap space of the jenkins server to 8GB by modifying the systemd service configuration this way:

software-dev@magnet:~$ sudo cat /etc/systemd/system/jenkins.service.d/override.conf
[Service]
Environment="JAVA_OPTS=-Djava.awt.headless=true -Xmx8g"

which was taken into account, because I verified with the following command:

software-dev@magnet:~$ tr '\0' '\n' < /proc/$(pidof java)/cmdline
/usr/bin/java
-Djava.awt.headless=true
-Xmx10g
-jar
/usr/share/java/jenkins.war
--webroot=/var/cache/jenkins/war
--httpPort=8080

I just increased the Heap size to 10GB and I'll wait for the result of this night's build, but I have the feeling that this amount of Heap space really looks excessive, so I'm suspecting that a plugin, maybe the JUnit one, may be buggy and could consume too much memory.

Is anyone aware of such a thing? Could there be workarounds?

More importantly, which methods could I use to try to track if one plugin is consuming too much? I have notions of Java since my CS degree, but I'm not familiar with the jenkins development ecosystem.

Thank you by advance.

Answer 1

You can try splitting the tests into chunks/batch/groups but this solution requires changes in the code.

More details

https://semaphoreci.com/community/tutorials/how-to-split-junit-tests-in-a-continuous-integration-environment

Grouping JUnit tests