JMeter Maven Distributed - Immediate Shutdown with 0 Results
Question
I am developing a JMeter dynamic master-slaves Performance Environment with the help of Azure Kubernetes, Jenkins, Helm, JMeter Java Plugin V3.0.0, JMeter version 5.2.1 . Jnlp, Maven and Helm are running inside a Jenkins Agent while the JMeter Slaves are spawned up as Kubernetes Deployment Pods. During the execution both the JMeter Master, Slave nodes are spawning and it will start the slave agents successfully as well. But after some time, I see it's shutting down immediately with 0 results.
I have tried disabling the Distributed Jmeter Slaves and it is working fine without any issue when executing only from the Master. Please help me to find out the issue with the slave configs
pom.xml File
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<name>Performance Setup Onboard</name>
<groupId>groupId</groupId>
<artifactId>perf-onboard</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>pom</packaging>
<properties>
<jmeter.version>5.2.1</jmeter.version>
<jmeter.maven.plugin.version>3.0.0</jmeter.maven.plugin.version>
<maven.build.timestamp.format>yyyyMMdd</maven.build.timestamp.format>
<!--<timestamp>${maven.build.timestamp}</timestamp>-->
<jmeter.directory>${basedir}/target/jmeter</jmeter.directory>
<jmeter.dashboard.dir>${jmeter.directory}/results/html</jmeter.dashboard.dir>
<report.generation>true</report.generation>
<jmxFile>testDemoScript</jmxFile>
<jenkinsSlaveNodes>1</jenkinsSlaveNodes>
</properties>
<build>
<plugins>
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>3.2.0</version>
<executions>
<execution>
<id>copy-test-data-1</id>
<phase>verify</phase>
<goals>
<goal>copy-resources</goal>
</goals>
<configuration>
<outputDirectory>${jmeter.directory}/testFiles/data</outputDirectory>
<resources>
<resource>
<directory>./src/test/data</directory>
<includes>
<include>*.csv</include>
</includes>
</resource>
</resources>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>com.lazerycode.jmeter</groupId>
<artifactId>jmeter-maven-plugin</artifactId>
<version>${jmeter.maven.plugin.version}</version>
<executions>
<execution>
<id>configuration</id>
<goals>
<goal>configure</goal>
</goals>
</execution>
<execution>
<id>jmeter-tests</id>
<phase>verify</phase>
<goals>
<goal>jmeter</goal>
</goals>
</execution>
<execution>
<id>jmeter-check-results</id>
<goals>
<goal>results</goal>
</goals>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>kg.apc</groupId>
<artifactId>jmeter-plugins-extras-libs</artifactId>
<version>1.4.0</version>
</dependency>
</dependencies>
<configuration>
<testFilesIncluded>
<jMeterTestFile>${jmxFile}.jmx</jMeterTestFile>
</testFilesIncluded>
<resultsFileNameDateFormat>${maven.build.timestamp.format}</resultsFileNameDateFormat>
<jMeterProcessJVMSettings>
<xms>1024</xms>
<xmx>1024</xmx>
<arguments>
<argument>-Xprof</argument>
<argument>-Xfuture</argument>
</arguments>
</jMeterProcessJVMSettings>
<ignoreResultFailures>true</ignoreResultFailures>
<remoteConfig>
<startServersBeforeTests>true</startServersBeforeTests>
<!--<stopServersAfterTests>false</stopServersAfterTests>-->
<serverList>${jenkinsSlaveNodes}</serverList>
</remoteConfig>
<propertiesJMeter>
<client.rmi.localport>1099</client.rmi.localport>
<!--<https.socket.protocols>TLSv1.2</https.socket.protocols>-->
</propertiesJMeter>
<propertiesUser>
<server.rmi.ssl.disable>true</server.rmi.ssl.disable>
<jmeter.save.saveservice.output_format>csv</jmeter.save.saveservice.output_format>
</propertiesUser>
<generateReports>${report.generation}</generateReports>
<propertiesGlobal>
<!--You can define your Customer JMeter Parameters if available-->
<protocol>${protocol}</protocol>
<host>${host}</host>
<basePath>${basePath}</basePath>
</propertiesGlobal>
<jmeterVersion>${jmeter.version}</jmeterVersion>
<jmeterExtensions>
<artifact>kg.apc:jmeter-plugins-extras:1.4.0</artifact>
<artifact>kg.apc:jmeter-plugins-standard:1.4.0</artifact>
<artifact>kg.apc:jmeter-plugins-autostop:0.1</artifact>
<artifact>kg.apc:jmeter-plugins-casutg:2.9</artifact>
<artifact>kg.apc:jmeter-plugins-cmn-jmeter:0.6</artifact>
<artifact>kg.apc:jmeter-plugins-csvars:0.1</artifact>
<artifact>kg.apc:jmeter-plugins-functions:2.1</artifact>
<artifact>kg.apc:jmeter-plugins-manager:1.3</artifact>
<artifact>kg.apc:jmeter-plugins-perfmon:2.1</artifact>
<artifact>kg.apc:jmeter-plugins-prmctl:0.4</artifact>
<artifact>kg.apc:jmeter-plugins-tst:2.5</artifact>
<artifact>kg.apc:jmeter-plugins-webdriver:3.1</artifact>
<artifact>kg.apc:jmeter-plugins-dummy:0.4</artifact>
<artifact>com.blazemeter:jmeter-parallel:0.9</artifact>
<artifactItem>kg.apc:jmeter-plugins-manager:1.3</artifactItem>
<artifactItem>kg.apc:jmeter-plugins-json:2.6</artifactItem>
</jmeterExtensions>
</configuration>
</plugin>
</plugins>
</build>
</project>
JMeter Master Yaml
apiVersion: v1
kind: Pod
metadata:
name: distributed-jmeter-master
namespace: perf-platform
labels:
app.kubernetes.io/instance: distributed-jmeter-master
spec:
containers:
- name: distributed-jmeter-master
image: maven:3.8.3-jdk-11-openj9
imagePullPolicy: IfNotPresent
command:
- cat
tty: true
resources:
requests:
memory: "1024Mi"
cpu: "100m"
limits:
memory: "2048Mi"
cpu: "200m"
volumeMounts:
- mountPath: /root/.m2
name: m2-volume
- name: kubehelm
image: dtzar/helm-kubectl:3.7.0
imagePullPolicy: IfNotPresent
command:
- cat
tty: true
resources:
requests:
memory: "256Mi"
cpu: "100m"
limits:
memory: "512Mi"
cpu: "200m"
volumeMounts:
- mountPath: /root/.config
name: helm-volume
volumes:
- name: m2-volume
persistentVolumeClaim:
claimName: m2-pvc
- name: helm-volume
persistentVolumeClaim:
claimName: helm-pvc
JMeter Slave Yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Values.dep.name }}
namespace: perf-platform
labels:
app.kubernetes.io/name: {{ .Values.dep.name }}
spec:
replicas: {{ .Values.slave.replicaCount }}
selector:
matchLabels:
app.kubernetes.io/name: {{ .Values.dep.name }}
template:
metadata:
labels:
app.kubernetes.io/name: {{ .Values.dep.name }}
spec:
containers:
- name: distributed-jmeter-slave
image: "{{ .Values.slave.image }}:{{ .Values.slave.tag }}"
imagePullPolicy: {{ .Values.slave.pullPolicy }}
env:
- name: MODE
value: "slave-mode"
- name: XMS_VALUE
value: "{{ .Values.slave.heap.xms.memory }}"
- name: XMX_VALUE
value: "{{ .Values.slave.heap.xmx.memory }}"
args: ["$(MODE)","$(XMS_VALUE)", "$(XMX_VALUE)"]
ports:
- containerPort: 50000
- containerPort: 1099
resources:
requests:
memory: "{{ .Values.slave.res.req.mem }}"
cpu: "{{ .Values.slave.res.req.cpu }}"
limits:
memory: "{{ .Values.slave.res.lim.mem }}"
cpu: "{{ .Values.slave.res.lim.cpu }}"
JMeter Slave Values Yaml File
#JMeter Slave Configuration
dep:
name: distributed-jmeter
slave:
replicaCount: 2
image: sjay8691/jmeter-all-plugins-slave
tag: 5.2.1_jdk11_v1
pullPolicy: IfNotPresent
res:
req:
mem: "1024Mi"
cpu: "100m"
lim:
mem: "1024Mi"
cpu: "100m"
heap:
xms1:
memory: 512M
xms2:
memory: 512M
Jenkins Pipeline
pipeline {
agent{
kubernetes {
yamlFile 'JMeter-master-pod.yaml'
}
}
environment {
JOBNAME = "jmeter-perf"
}
parameters {
//Project onboarding - Set JMeter Parameters
string(name: 'jmxFile', description: 'scriptName', defaultValue: "testDemoScript")
string(name: 'protocol', description: 'protocol', defaultValue: "https")
string(name: 'host', description: 'host', defaultValue: "catfact.ninja")
string(name: 'basePath', description: 'basePath', defaultValue: "/breeds?limit=")
string(name: 'testDataFile', description: 'testDataFile', defaultValue: "testData.csv")
}
stages {
stage ('Deploy JMeter Master-Slave Slaves'){
steps{
container ('kubehelm') {
sh 'echo =========================================================== Start deploying JMeter [Slave] Pods ==========================================================='
sh 'helm repo update'
sh 'helm install --wait jmeter-${JOBNAME}-${BUILD_NUMBER} --set dep.name=jmeter-distributed-slave-${JOBNAME}-${BUILD_NUMBER} custom/10-distributed-jmeter -f JMeter-slave-pods.yaml'
sh 'kubectl wait --for=condition=ready pods -l app.kubernetes.io/name=jmeter-distributed-slave-${JOBNAME}-${BUILD_NUMBER} -n perf-platform --timeout=90s'
sh 'echo =========================================================== Completed deploying JMeter [Master - Slave] Nodes ==========================================================='
}
}
}
stage ('Get Slave Pods IP Addresses') {
steps {
container ('kubehelm') {
sh 'echo =========================================================== Start searching JMeter [Slave] IP Addresses ==========================================================='
script {
env.jenkinsSlaveNodes = sh(returnStdout: true, script:'kubectl get pods -l app.kubernetes.io/name=jmeter-distributed-slave-${JOBNAME}-${BUILD_NUMBER} -n perf-platform -o jsonpath=\'{.items[*].status.podIP}\' | tr \' \' \',\'')
}
sh "echo =========================================================================================================="
sh "echo Slave IP Details : [${env.jenkinsSlaveNodes}]"
sh "echo =========================================================================================================="
sh 'echo =============================================s============== Finish searching JMeter [Slave] IP Addresses ==========================================================='
}
}
}
stage ('Copy Test Data Files') {
steps {
container ('kubehelm') {
sh 'echo =========================================================== Start copying Test Data Files into JMeter [Slave] Pods ==========================================================='
script {
env.podNames = sh(returnStdout: true, script:'kubectl get pods -l app.kubernetes.io/name=jmeter-distributed-slave-${JOBNAME}-${BUILD_NUMBER} -n perf-platform -o jsonpath=\'{.items[*].metadata.name}\' | tr \' \' \',\'')
}
sh '''#!/bin/bash
IFS=',' read -r -a podList <<< "$podNames"
for pod in "${podList[@]}"
do
echo "================================================================================================"
echo "Copying data files to : $pod [POD]"
kubectl cp ./src/test/data perf-platform/$pod:./bin/data
echo "================================================================================================"
done
'''
sh "echo Slave Pod Names : ${env.podNames}"
sh 'echo =========================================================== Finish Copying Test Data Files into JMeter [Slave] Pods ==========================================================='
}
}
}
stage ('Execute JMeter Performance Test') {
steps {
container ('distributed-jmeter-master') {
sh 'echo =========================================================== Start JMeter Performance Test Execution ==========================================================='
sh '''
mvn clean install -DjenkinsSlaveNodes=${jenkinsSlaveNodes} -DjmxFile=${jmxFile} -Dprotocol=${protocol} \
-Dhost=${host} -DbasePath=${basePath} -DtestDataFile=${testDataFile} \
'''
sh 'echo =========================================================== Completed JMeter Performance Test Execution ==========================================================='
}
}
}
stage ('Publish JMeter Test Results') {
steps {
sh 'echo =========================================================== Start publishing JMeter Performance Test Results ==========================================================='
sh "echo Present Working Directory : pwd"
perfReport 'target/jmeter/results/*.csv'
sh 'echo =========================================================== Finish publishing JMeter Performance Test Results ==========================================================='
}
}
}
post {
always {
sh 'echo =========================================================== Publishing JMeter Log ==========================================================='
publishHTML (target : [
allowMissing: false,
alwaysLinkToLastBuild: true,
keepAll: true,
reportDir: "target/jmeter/logs/",
reportFiles: "${jmxFile}.jmx.log",
reportName: 'JMeter Log File']
)
sh 'echo =========================================================== Finish publishing JMeter Log ==========================================================='
container ('kubehelm') {
sh 'echo =========================================================== Start erasing JMeter [Slave] Pods ==========================================================='
sh 'helm uninstall jmeter-${JOBNAME}-${BUILD_NUMBER}'
sh 'kubectl wait --for=delete pods -l app.kubernetes.io/name=jmeter-distributed-slave-${JOBNAME}-${BUILD_NUMBER} -n perf-platform --timeout=90s'
sh 'echo =========================================================== Finish erasing JMeter [Master - Slave] Nodes ==========================================================='
}
archiveArtifacts artifacts: 'target/**', fingerprint: true
}
}
}
Jenkins Log
Downloading dependencies: true
[INFO] Configuring JMeter properties...
[INFO]
[INFO] --- maven-resources-plugin:3.2.0:copy-resources (copy-test-data-1) @ perf-onboard ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Using 'null' encoding to copy filtered properties files.
[INFO] Copying 1 resource
[INFO]
[INFO] --- jmeter-maven-plugin:3.0.0:jmeter (jmeter-tests) @ perf-onboard ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] P E R F O R M A N C E T E S T S
[INFO] -------------------------------------------------------
[INFO]
[INFO] Will generate HTML report in /home/jenkins/agent/workspace/jmeter-perf-dynamic-slaves/target/jmeter/reports/testDemoScript
[INFO] Executing test: testDemoScript.jmx
[INFO] Arguments for forked JMeter JVM: [java, -Xms1024M, -Xmx1024M, -Xprof, -Xfuture, -Djava.awt.headless=true, -jar, ApacheJMeter-5.2.1.jar, -d, /home/jenkins/agent/workspace/jmeter-perf-dynamic-slaves/target/2243f4e2-f010-44d4-96bd-da99fa32fdaa/jmeter, -e, -j, /home/jenkins/agent/workspace/jmeter-perf-dynamic-slaves/target/jmeter/logs/testDemoScript.jmx.log, -l, /home/jenkins/agent/workspace/jmeter-perf-dynamic-slaves/target/jmeter/results/20211008-testDemoScript.csv, -n, -o, /home/jenkins/agent/workspace/jmeter-perf-dynamic-slaves/target/jmeter/reports/testDemoScript, -r, -t, /home/jenkins/agent/workspace/jmeter-perf-dynamic-slaves/target/jmeter/testFiles/testDemoScript.jmx, -R, 10.240.0.215, -Dsun.net.http.allowRestrictedHeaders, true, -Gprotocol=https, -GbasePath=/breeds?limit=, -Ghost=catfact.ninja]
[INFO]
[INFO] SLF4J: Class path contains multiple SLF4J bindings.
[INFO] SLF4J: Found binding in [jar:file:/home/jenkins/agent/workspace/jmeter-perf-dynamic-slaves/target/2243f4e2-f010-44d4-96bd-da99fa32fdaa/jmeter/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
[INFO] SLF4J: Found binding in [jar:file:/home/jenkins/agent/workspace/jmeter-perf-dynamic-slaves/target/2243f4e2-f010-44d4-96bd-da99fa32fdaa/jmeter/lib/slf4j-nop-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
[INFO] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
[INFO] SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
[INFO] Oct 08, 2021 9:22:39 AM java.util.prefs.FileSystemPreferences$1 run
[INFO] INFO: Created user preferences directory.
[INFO] WARNING: An illegal reflective access operation has occurred
[INFO] WARNING: Illegal reflective access by com.thoughtworks.xstream.core.util.Fields (file:/home/jenkins/agent/workspace/jmeter-perf-dynamic-slaves/target/2243f4e2-f010-44d4-96bd-da99fa32fdaa/jmeter/lib/xstream-1.4.11.jar) to field java.util.TreeMap.comparator
[INFO] WARNING: Please consider reporting this to the maintainers of com.thoughtworks.xstream.core.util.Fields
[INFO] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
[INFO] WARNING: All illegal access operations will be denied in a future release
[INFO] Creating summariser <summary>
[INFO] Created the tree successfully using /home/jenkins/agent/workspace/jmeter-perf-dynamic-slaves/target/jmeter/testFiles/testDemoScript.jmx
[INFO] Configuring remote engine: 10.240.0.215
[INFO] Starting distributed test with remote engines: [10.240.0.215] @ Fri Oct 08 09:22:44 GMT 2021 (1633684964936)
[INFO] Warning: Nashorn engine is planned to be removed from a future JDK release
[INFO] Remote engines have been started:[10.240.0.215]
[INFO] Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
[INFO] summary = 0 in 00:00:00 = ******/s Avg: 0 Min: 9223372036854775807 Max: -9223372036854775808 Err: 0 (0.00%)
[INFO] Tidying up remote @ Fri Oct 08 09:23:06 GMT 2021 (1633684986352)
[INFO] Error generating the report: java.lang.NullPointerException
[INFO] ... end of run
[INFO] Completed Test: /home/jenkins/agent/workspace/jmeter-perf-dynamic-slaves/target/jmeter/testFiles/testDemoScript.jmx
[INFO]
[INFO]
[INFO] --- jmeter-maven-plugin:3.0.0:results (jmeter-check-results) @ perf-onboard ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] S C A N N I N G F O R R E S U L T S
[INFO] -------------------------------------------------------
[INFO]
[INFO] Will scan results using format: CSV
[INFO]
[INFO] Parsing results file '/home/jenkins/agent/workspace/jmeter-perf-dynamic-slaves/target/jmeter/results/20211008-testDemoScript.csv' as type: CSV
[INFO] Number of failures in '20211008-testDemoScript.csv': 0
[INFO] Number of successes in '20211008-testDemoScript.csv': 0
[INFO]
[INFO] -------------------------------------------------------
[INFO] P E R F O R M A N C E T E S T R E S U L T S
[INFO] -------------------------------------------------------
[INFO]
[INFO] Result (.csv) files scanned: 1
[INFO] Successful requests: 0
[INFO] Failed requests: 0
[INFO] Failures: NaN% (0.0% accepted)
[INFO]
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ perf-onboard ---
[INFO] Installing /home/jenkins/agent/workspace/jmeter-perf-dynamic-slaves/pom.xml to /root/.m2/repository/groupId/perf-onboard/1.0-SNAPSHOT/perf-onboard-1.0-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
**JMeter Log**
2021-10-08 09:23:12,638 INFO o.a.j.r.d.JsonExporter: Creating statistics for overall
2021-10-08 09:23:12,638 ERROR o.a.j.JMeter: Error generating the report: null
java.lang.NullPointerException: null
at org.apache.jmeter.report.dashboard.JsonExporter.createStatistic(JsonExporter.java:121) ~[ApacheJMeter_core-5.2.1.jar:5.2.1]
at org.apache.jmeter.report.dashboard.JsonExporter.export(JsonExporter.java:71) ~[ApacheJMeter_core-5.2.1.jar:5.2.1]
at org.apache.jmeter.report.dashboard.ReportGenerator.exportData(ReportGenerator.java:377) ~[ApacheJMeter_core-5.2.1.jar:5.2.1]
at org.apache.jmeter.report.dashboard.ReportGenerator.generate(ReportGenerator.java:259) ~[ApacheJMeter_core-5.2.1.jar:5.2.1]
at org.apache.jmeter.JMeter$ListenToTest.endTest(JMeter.java:1330) [ApacheJMeter_core-5.2.1.jar:5.2.1]
at org.apache.jmeter.JMeter$ListenToTest.lambda$testEnded$0(JMeter.java:1276) [ApacheJMeter_core-5.2.1.jar:5.2.1]
at org.apache.jmeter.JMeter$ListenToTest$$Lambda$231/0x000000005814b440.run(Unknown Source) [ApacheJMeter_core-5.2.1.jar:5.2.1]
at java.lang.Thread.run(Thread.java:836) [?:?]
Kubernetes Pods enter image description here
1 answers
solution1
0 2021-10-25 12:27:48
You are running an old version of the jmeter-maven-plugin (3.0.0), so I suspect you may be suffering from this bug:
https://github.com/jmeter-maven-plugin/jmeter-maven-plugin/issues/356
When you run:
mvn com.lazerycode.jmeter:jmeter-maven-plugin:configure com.lazerycode.jmeter:jmeter-maven-plugin:remote-server
It appears to work and shows a build success, but in reality it's failing silently because there is no default rmi_keystore.jks file in the bin directory.
I would suggest increasing your jmeter-maven-plugin version up to the current version (3.5.0 at the time of writing this).
You can see the changes made between 3.0.0 and 3.5.0 by looking at the CHANGELOG .
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
Conditionally Execute JMeter Maven Plugin
Setup & Execution of Jmeter Distributed Test run from Jenkins
How to configure Jenkins for distributed load using multiple Jmeter-servers
Running jmeter through jenkins with maven/ant
How to run JMeter tests using Maven
Unable to save results in results directory using jmeter with jenkins
JMeter: Stoptest.sh/Shutdown.sh doesn't work from Jenkins
Maven plugin to export junit results
Jmeter(maven) doesn't run on jenkins but runs locally
Maven error in Jenkins JMeter project, “Error creating properties files for forking”
Related Tags