JMeter/Azure/Taurus - Errors Observed When Running Tests On Multiple Nodes/Workers

Question

We are running tests as part of an Azure pipeline using Taurus. We have a python scripts that scales up to 10 nodes (Node size: standard_B8ms) and 10 JMeter workers. The JMeter scripts report the results to Grafana. Whilst running the tests we can see the (Grafana, Reporting and JMeter) processes on the Nodes.

The issue being observed is when running the test locally with 5 threads, using different logins, the script works fine with no errors. However, we scale the test up to 35 users across 5 nodes and 5 workers there are errors observed across the transactions. Has this got anything to do with using multiple nodes? We are using the "bzm - Random CSV Data Set Config" plugin to ensure didn't users login into the application across the nodes whilst the users ramp up.

There were no issues from the node resource perspective. The CPU would hit a maximum of 10% during the ramp up and drop down after that.

Has anyone else observed this? If so, what is an effective solution to improve the reliability of the JMeter tests? Is it better to use one of the Custom Thread Groups to control the ramp up and hold periods?

Thank you.

Answer 1

If the issue cannot be reproduced on lower loads (5 threads) and is reproduced for higher loads (35 threads) it's definitely somehow connected with the fact the system under test gets overloaded and cannot handle such amount of virtual users.

Also be informed that when you run your test in distributed mode the number of threads you set in the Thread Group is being multiplied by the number of JMeter slaves so:

• if you have 5 threads and 10 slaves - there are 50 virtual users in total
• if you have 35 threads and 10 slaves - there are 350 virtual users in total

So check the following:

• error.jtl file in Taurus Artifacts directory
• jmeter.log file on each slave machine, if it doesn't tell anything meaningful increase JMeter logging verbosity to DEBUG level
• system under test log files