SQL Server query performance slows over time

Question

I've seen this question asked in many ways all over the Internet but despite implementing the abundance of advice (and some voodoo), I'm still struggling. I have a 100GB+ database that is constantly inserting and updating records in very large transactions (200+ statements per trans). After a system restart, the performance is amazing (data is written to a large SATA III SSD connected via USB 3.0). The SQL Server instance is running on a VM running under VMWare Workstation. The host is set to hold the entire VM in memory. The VM itself has a paging cache of 5000 MB. The SQL Server user is set to 'hold pages in memory'. I have 5 GBs of RAM allocated to the VM, and the max memory of the SQL Server instance is set to half a Gig.

I have played with every single one of these parameters to attempt to maintain consistent performance, but sure and steady, the performance eventually degrades to the point where it begins to time out. Here's the kicker though, if I stop the application that's loading the database, and then execute the stored proc in the Management Studio, it runs like lightning, clearly indicating it's not an issue with the query, and probably nothing to do with memory management or paging. If I then restart the loader app, it still crawls. If I reboot the VM however, the app once again runs like lightning...for a while...

Does anybody have any other suggestions based upon the symptoms presented?

Answer 1

• Depending on how large your hot set is, 5GB memory may just tax it for a 100+gb database.

• Check indices and query plans. We can not help you without them. And I bet you miss some indices - which is the standard performance issue people have.

• Otherwise, once you made your homework - head over to dba.stackexchange.com and ask there.

• Generally - consider that 200 statements per transaction may simply indicate a seriously sub-optimal programming. For example you could bulk-load the data into a temp table then merge into the final one.

Answer 2

Actually, I may have a working theory. What I did was add some logic to the app that when it times out, sit for two minutes, and then try again, and voila! Back to full speed. I rubber-ducky'd my co-worker and came up with the concept that my perceived SSD write speeds were actually the write speed to the VMWare host's virtual USB 3 buffer, and that the actual SSD write speeds were slower. I'm probably hitting against the host's buffer size and by forcing the app to wait 2 minutes, the host has a chance to dump its back-buffered data to the SSD. Elementary, Watson :)

If this approach also fails to be sustainable, I'll report in.

Answer 3

Try executing this to determine your problem queries:

SELECT TOP 20
  qs.sql_handle,
  qs.execution_count,
  qs.total_worker_time AS Total_CPU,
  total_CPU_inSeconds = --Converted from microseconds
      qs.total_worker_time/1000000,
  average_CPU_inSeconds = --Converted from microseconds
      (qs.total_worker_time/1000000) / qs.execution_count,
  qs.total_elapsed_time,
  total_elapsed_time_inSeconds = --Converted from microseconds
      qs.total_elapsed_time/1000000,
 st.text,
 qp.query_plan
FROM
 sys.dm_exec_query_stats as qs
 CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) as st
 cross apply sys.dm_exec_query_plan (qs.plan_handle) as qp
 ORDER BY qs.total_worker_time desc

Then check your estimated and actual execution plans on the queries this command helps you pinpoint.

Source How do I find out what is hammering my SQL Server? and at the bottom of the page of http://technet.microsoft.com/en-us/magazine/2007.11.sqlquery.aspx

Answer 4

Beyond the excellent indexing suggestions already given, be sure to read up on parameter sniffing. That could be the cause of the problem.

SQL Server - parameter sniffing

http://www.sommarskog.se/query-plan-mysteries.html#compileps

As a result you could have a bad query plan being re-used, or SQL's buffer could be getting full and writing pages out of memory to disk (maybe that's other allocated memory in your case).

You could run DBCC FreeProcCache and DBCC FreeSystemCache to empty it and see if you get a performance boost.

You should give SQL more memory too - as much as you can while leaving room for other critical programs and the OS. You might have 5gb of Ram on the VM, but SQL is only getting to play with a 1/2 gb, which seems REALLY small for what you're describing.

If those things don't move you in the right direction, install the SQL Management Data Warehouse so you can see exactly what is happening when your slow down begins. Running it takes up additional memory, but you will give the DBA's more to go on.

Answer 5

In the end, what I did was a combination of two things, putting in logic to recover when timeouts occurred, and setting the host core count to only reflect physical cores, not logical cores, so for example, the host has 2 cores that are hyper-threaded. When I set my VM to use 4 cores, it occasionally gets hung in some infinite loop, but when I set it to 2 cores, it runs without fail. Still, aberrant behavior like this is difficult to mitigate reliably.