Jmeter bottleneck

Question

We have a system which we are testing using Jmeter. We experience the following bottleneck in our Jmeter ramp-up test:

Symptoms:

• We reach a maximum of 15 requests per second (rps).
• This limit is per Java VM, not per computer.
• CPU is not at its limit, only at ca. 30%. The test does quite some pre- and post-processing together with the requests, so this could have been a problem, but isn't.
• Memory (computer with 3.5 GiB) is at ca. 70% - 80%. This is sufficient to go somewhat higher with the rps, so this is not the limit.
• Memory (JVM) has been set according to the recommendations, and also been varied to experiment with it, but no improvement.
• I also checked the TCP connections, a change from “use keep-alive” to “don't use keep-alive” hasn't improved anything (was not to be expected).
• All(!) assertions are switched off.
• Logging: Only response times, response times percentiles, throughput, and request rate are logged, together with the standard table output of jmeter.
• All log files are saved to the local hard disk.

Jmeter is started over a jmeter.bat script with thefollowing settings for the JVM:

set HEAP=-Xms768m -Xmx768m -Xss128k
set NEW=-XX:NewSize=256m -XX:MaxNewSize=256m
set SURVIVOR=-XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=50%
set TENURING=-XX:MaxTenuringThreshold=2
set RMIGC=-Dsun.rmi.dgc.client.gcInterval=600000 -Dsun.rmi.dgc.server.gcInterval=600000
set PERM=-XX:PermSize=256m -XX:MaxPermSize=256m
set DUMP=-XX:+HeapDumpOnOutOfMemoryError
set DDRAW=
set ARGS=%DUMP% %HEAP% %NEW% %SURVIVOR% %TENURING% %RMIGC% %PERM% %DDRAW% %JM_START% %JM_LAUNCH% %ARGS% %JVM_ARGS% -jar "%JMETER_BIN%ApacheJMeter.jar" %JMETER_CMD_LINE_ARGS%

jmeter.properties has the following content:

language=en
cookies=cookies
xml.parser=org.apache.xerces.parsers.SAXParser
jmeter.laf.mac=System
jmeter.loggerpanel.maxlength=5000
not_in_menu=HTML Parameter Mask,HTTP User Parameter Modifier
remote_hosts=127.0.0.1
log_level.jmeter=ERROR
log_level.jmeter.junit=ERROR
log_level.jorphan=WARN
log_file='c:\\temp\\jmeter_'yyyyMMdd'.log'
jmeter.save.saveservice.timestamp_format=yyyy/MM/dd HH:mm:ss
jmeter.save.saveservice.base_prefix=~/
sampleresult.timestamp.start=true
upgrade_properties=/bin/upgrade.properties
HTTPResponse.parsers=htmlParser wmlParser
htmlParser.types=text/html application/xhtml+xml application/xml text/xml
wmlParser.className=org.apache.jmeter.protocol.http.parser.RegexpHTMLParser
wmlParser.types=text/vnd.wap.wml
summariser.interval=180
summariser.log=true
summariser.out=true
beanshell.server.file=../extras/startup.bsh
time.YMDHMS=yyyyMMdd_HHmmss
onload.expandtree=true
view.results.tree.max_size=0
classfinder.functions.contain=.functions.
classfinder.functions.notContain=.gui.
user.properties=user.properties
system.properties=system.properties   

At the jmx the following settings have been specified:

<stringProp name="ThreadGroup.num_threads">300</stringProp>
<stringProp name="ThreadGroup.ramp_time">0</stringProp>
<stringProp name="ThreadGroup.ramp_time">0</stringProp>
<longProp name="ThreadGroup.start_time">1404986400000</longProp>
<longProp name="ThreadGroup.end_time">1404990000000</longProp>
<boolProp name="ThreadGroup.scheduler">false</boolProp>
<stringProp name="ThreadGroup.duration">3600</stringProp>
<stringProp name="ThreadGroup.delay"></stringProp>
<boolProp name="ThreadGroup.delayedStart">true</boolProp>

Question : Where is the bottleneck, and how can we get rid of it?

Answer 1

I would like to suggest few things,

1. Implement all JMeter best practices specified by Blazemerter JMeter tips (same link provided by Dmitri) This will make sure that scripts are optimized.
2. You have computer with 3.5 GB or ram but your JMeter setting shows, you have provided only 768mb max memory of JVM. Are you starting multiple instances of JMeter on same computer? I think single instance of JMeter with 2gb max heap size (or make 2 instances for mmgt purpose) should be able to produce 300 users load.
3. I see ramp up time is 0 for your test though test is running for 1 hr. for 300 users, all coming online in 0 seconds that will overload the system at the start. Is it really required? Probably you should add some ramp up of 10-15 mins and run the test, that will help improve performance of Jmeter instance.
4. Decide how much tps or req/sec you want to generate if it is present in your load testing plan(or SLA's) then probably use throughtput controller to control the required throughput.

5. Start Monitoring JMeter Instance using,

   JVisualVM : UI tool, freeware, good details, easy to use
   JMap : both UI and commandline, excellent details, easy to use
   Jstat : command line, very low overhead on system, excellent details

   and monitor application system under load as well for better analysis. This will help you to find application side bottleneck(if any) as well.

Generally normal user machine will also have 8gb ram, dual/quad core cpus and thus it is possible to produce 500-1000 user load from a single machine. If at all required, go with distributed JMeter testing.

Answer 2

Consider the following checklist:

1. non-GUI mode is a must
2. All Listeners need to be disabled
3. If you use any Beanshell test elements - install groovy scripting engine and switch to JSR223 Sampler and groovy language
4. See JMeter Performance and Tuning Tips guide for recommended JVM and GC tuning

If all above doesn't help - consider remote (distributed) testing