Docker container doesn't finish with with HTTP POST audio file in JMeter

Question

I am running load test using JMeter, the test sends a POST request with an audio file to the server and receives a response. I chose to go with docker on a Linux VM as moving forward I will need to do Distributed testing and thought it might easier to execute with Docker. When I use 1hr audio file everything seems to work fine except the fact that sometimes Jmeter executes more threads than scheduled. However if I use a larger file like 3h or 5h the container doesn't finish and exit even though I see on the server side that file is done processing for over 10 min. For the task I use modified Dockerfile and image that I found on dockerhub / git hub "justb4/jmeter". The Dockerfile as follows:

# inspired by https://github.com/hauptmedia/docker-jmeter  and
# https://github.com/hhcordero/docker-jmeter-server/blob/master/Dockerfile
FROM alpine:3.12

MAINTAINER Just van den Broecke<just@justobjects.nl>
# modified by Weronika Siwak

ARG JMETER_VERSION="5.4.3"

ENV JMETER_HOME /opt/apache-jmeter-${JMETER_VERSION}
ENV JMETER_BIN  ${JMETER_HOME}/bin
ENV JMETER_DOWNLOAD_URL  https://archive.apache.org/dist/jmeter/binaries/apache- 
jmeter-${JMETER_VERSION}.tgz
ENV JMETER_PLUGINS_FOLDER ${JMETER_HOME}/lib/ext/



# Install extra packages
# Set TimeZone, See: https://github.com/gliderlabs/docker-alpine/issues/136#issuecomment- 
612751142
ARG TZ="America/Chicago"
ENV TZ ${TZ}
RUN    apk update \
&& apk upgrade \
&& apk add ca-certificates \
&& update-ca-certificates \
&& apk add --update openjdk8-jre tzdata curl unzip bash \
&& apk add --no-cache nss \
&& rm -rf /var/cache/apk/* \
&& mkdir -p /tmp/dependencies  \
&& curl -L --silent ${JMETER_DOWNLOAD_URL} >  /tmp/dependencies/apache-jmeter-${JMETER_VERSION}.tgz  \
&& mkdir -p /opt  \
&& tar -xzf /tmp/dependencies/apache-jmeter-${JMETER_VERSION}.tgz -C /opt  \
&& rm -rf /tmp/dependencies \
&& mkdir -p /opt/apache-jmeter-${JMETER_VERSION}/bin/test-plans \
&& mkdir -p /opt/apache-jmeter-${JMETER_VERSION}/bin/audio






RUN    wget https://jmeter-plugins.org/files/packages/jpgc-graphs-basic-2.0.zip \
  && unzip jpgc-graphs-basic-2.0.zip -d ${JMETER_HOME} \
  && rm jpgc-graphs-basic-2.0.zip \
  && wget https://jmeter-plugins.org/files/packages/jpgc-graphs-additional-2.0.zip \
  && unzip -n jpgc-graphs-additional-2.0.zip -d ${JMETER_HOME} \ 
  && rm jpgc-graphs-additional-2.0.zip \
  && wget https://jmeter-plugins.org/files/packages/jpgc-cmd-2.2.zip \
  && unzip -n jpgc-cmd-2.2.zip -d ${JMETER_HOME} \
  && rm jpgc-cmd-2.2.zip \
  && wget https://jmeter-plugins.org/files/packages/jpgc-casutg-2.10.zip \
  && unzip -n jpgc-casutg-2.10.zip  -d ${JMETER_HOME}\
  && rm jpgc-casutg-2.10.zip \
  && wget https://jmeter-plugins.org/files/packages/jpgc-filterresults-2.2.zip \
  && unzip -n jpgc-filterresults-2.2.zip -d ${JMETER_HOME} \
  && rm jpgc-filterresults-2.2.zip \
  && wget https://jmeter-plugins.org/files/packages/jpgc-ggl-2.0.zip \
  && unzip -n jpgc-ggl-2.0.zip -d ${JMETER_HOME}\
  && rm jpgc-ggl-2.0.zip \
  && wget https://jmeter-plugins.org/files/packages/jmeter.pack-listener-1.7.zip \
  && unzip -n jmeter.pack-listener-1.7.zip -d ${JMETER_HOME}\
  && rm jmeter.pack-listener-1.7.zip \
  && wget https://jmeter-plugins.org/files/packages/bzm-parallel-0.11.zip \
  && unzip -n bzm-parallel-0.11.zip  -d ${JMETER_HOME}\
  && rm bzm-parallel-0.11.zip \ 
  && wget https://jmeter-plugins.org/files/packages/jpgc-perfmon-2.1.zip \
  && unzip -n jpgc-perfmon-2.1.zip -d ${JMETER_HOME} \
  && rm jpgc-perfmon-2.1.zip \
  && wget https://jmeter-plugins.org/files/packages/jpgc-synthesis-2.2.zip \
  && unzip -n jpgc-synthesis-2.2.zip -d ${JMETER_HOME} \
  && rm jpgc-synthesis-2.2.zip 

 # TODO: plugins (later)
 # && unzip -oq "/tmp/dependencies/JMeterPlugins-*.zip" -d $JMETER_HOME

 # Set global PATH such that "jmeter" command is found
 ENV PATH $PATH:$JMETER_BIN

# Entrypoint has same signature as "jmeter" command
COPY entrypoint.sh /


WORKDIR ${JMETER_HOME}
RUN ["chmod", "+x", "/entrypoint.sh"]

ENTRYPOINT [ "/entrypoint.sh"]

The entrypoint.sh:

#!/bin/bash
# Inspired from https://github.com/hhcordero/docker-jmeter-client
# Basically runs jmeter, assuming the PATH is set to point to JMeter bin-dir (see Dockerfile)
#
# This script expects the standdard JMeter command parameters.
#

# Install jmeter plugins available on /plugins volume
if [ -d /plugins ]
then
for plugin in /plugins/*.jar; do
    cp $plugin $(pwd)/lib/ext
done;
fi

# Execute JMeter command
set -e
freeMem=`awk '/MemFree/ { print int($2/1024) }' /proc/meminfo`
s=$(($freeMem/10*8))
x=$(($freeMem/10*8))
n=$(($freeMem/10*2))
export JVM_ARGS="-Xmn${n}m -Xms${s}m -Xmx${x}m"

echo "START Running Jmeter on `date`"
echo "JVM_ARGS=${JVM_ARGS}"
echo "jmeter args=$@"

# Keep entrypoint simple: we must pass the standard JMeter arguments
EXTRA_ARGS=-Dlog4j2.formatMsgNoLookups=true
echo "jmeter ALL ARGS=${EXTRA_ARGS} $@"
jmeter ${EXTRA_ARGS} $@

echo "END Running Jmeter on `date`"

#     -n \
#    -t "/tests/${TEST_DIR}/${TEST_PLAN}.jmx" \
#    -l "/tests/${TEST_DIR}/${TEST_PLAN}.jtl"
# exec tail -f jmeter.log
#    -D "java.rmi.server.hostname=${IP}" \
#    -D "client.rmi.localport=${RMI_PORT}" \
#  -R $REMOTE_HOSTS

For tests and results I use volumes, I execute with commands: jmeter -n -t bin/test-plans/1_usr_3_hr_15n.jmx -l /opt/apache-jmeter-5.4.3/bin/results/1_usr_3_hr_15n/1_usr_3_hr_15n.jtl -e -o /opt/apache-jmeter-5.4.3/bin/results/1_usr_3_hr_15n I don't know why it works for 1hr audio but not larger and why it executes more threads than scheduled. The test plan is simple 1 post http request with no loops, 1 thread per second

Answer 1

Docker neither solves the problem of JMeter execution nor makes distributed execution easier (especially if you're running everything at the same physical or virtual machine), it just consumes resources and being yet another layer when errors can occur

If JMeter test execution doesn't finish in the anticipated time I can think of the following reasons and steps to take:

1. The server fails to respond. By default JMeter waits for the response forever (unless limited by underlying OS or JVM timeouts) so to avoid "hanging" of the test in case when server fails to provide a response I would recommend setting a reasonable timeout, it can be done under "Advanced" tab of the HTTP Request sampler (or better HTTP Request Defaults )

   

2. Check jmeter.log file for any suspicious entries

3. You can log into slave containers and take thread dumps to see what exactly threads are doing and where/why they're stuck