SQL Server 2008 slow down after 30,000 user Insert from a busy Java web application

Question

One of our system finally launch in production and during the peak hour (about 200 concurrent user at any one time), traffic can ramp up to 30,000 user transaction within an hour.

What we notice is a strange behavior that right after our SQL server restarted, the performance is very fast. Even with 200 concurrent user at the beginning, the transaction to SQL server 2008 R2 is less than 10ms. However, after about 15,000 transactions, we can see each transaction can take up 100ms to complete. When comes to about 30,000 transactions, each transaction may take up 300ms the worst case. If we dont restart the SQL server, even single thread, the transaction will still be around 300ms.

Each transaction will perform the following:

• 1 Select in Master table and and 2 select in account table, 1 insert or update into master table, 1 insert or update to account table, and 1 or 2 insert into history table
• account table (2 column as primary key, 4 column as index and another 6 column as data column)
• master table (3 column as primary key, 5 column as index and another 11 column as data column)
• history table (3 column as primary key, 4 column as index and another 15 column as data column)

Note: Some data column above is index above is because we used in the where clause.

Note: During Select, we does not join any table for performance purposes.

Note: DB Server and Web Server is actually the same server.

System setup:

• OS: Windows 2008 Enterprise
• Web Server: Tomcat Web Server
• Connection to DB: Spring Connection Pooling (Min 50, Max 350)
• Using the Latest SQL JDBC Driver
• DB Server: Sql Server 2008 R2 SP1

Server Hardware:

• CPU: 4 x Quad Core CPU (each core have 2 threads) that means total of 32 physical threads
• Memory: 16GB
• Harddisk: raid 1 on 4 x SAS 15K Harddisk. this means 2 harddisk visible for OS. Application data and OS use 1 harddisk while SQL DB use another harddisk.

Any idea where we can track and resolve the performance slow down after 10,000 transactions?

Answer 1

It turns out the harddisk Raid configuration is wrong. Nothing to do with choosing either SQL/Oracle here, main culprit was the "Raid 1" configuration .

We delay the actual launch date and re-test with two harddisk on Raid-0 and it seems the speed of SQL server is much much better (30% faster) and is more consistence. Also, we re-format the harddisk to 64k cluster size which is recommended by the SQL MVP.

We ended up with 6 harddisks configured to Raid 1-0 in the production box and it runs like a champ!

Problem solved!