Microsoft Visual Studio Web Performance Test - Test result of first test always takes the longest?

Question

I have been running the web performance testing (Create Invoice) every day, and I noticed a trend that every time the tests were carried out for a number of iteration, the first result will always be the outlier (the one that takes the longest time) in terms of request time and total time. is there a way to prevent this from happening?

If browser caching should be taken into consideration. Which is the best solution?

• Run the test for once, but after once, record the result. Then clean the cache, re-run the test again and record the result. Run multiple times based on the requirement.
• Ignore the first test result, just take the remaining.

Answer 1

Sounds like this may be server side caching. I've seen this behavior a lot on systems I've load tested and consider it a valid but improbable result.

Overnight maintenance jobs or application updates can empty out back end database caches. The first user to log in the next morning will run into this situation where the system will need to do a full search through the database tables to retrieve the requested information instead of retrieving from the cache.

In my opinion this is a valid measurement. Just be mindful of how you present it. I like to keep it separate from averages. I'll give a cold start time and a warmed up average time.

Answer 2

Web Performance Tests should take browser caching into account. The first time a page is loaded with a normal browser all the dependent requests, (ie images, CSS, JavaScript, etc) must also be requested from the server. When the page is reloaded the browser will find many of the dependent requests can be found in the cache.

In a Visual Studio Web Performance Test each virtual user (VU) acts as if it has its own browser. When the VU starts it has an empty cache and so all pages and dependent requests are loaded. On the second and later executions by the same VU anything in the cache is not reloaded. The scenario property "percentage of new users" controls when new VUs are created and, effectively, when the cache is emptied.

Caching can be disabled. First by setting the Cache control property on the requests. Additionally caching of dependent requests can also be disabled by setting their Cache property to false. See page 186 of the Visual Studio Performance Testing Quick Reference Guide . There are many other useful sections about cache control in the document.

Neither suggestion in the question for handling the cache works for Visual Studio load tests. One method is to set the "percentage" of new users" in the load test scenario to 100%. That way each simulated user starts with an empty cache.