Improving search performance in large data sets

Question

On a WPF application already in production, users have a window where they choose a client. It shows a list with all the clients and a TextBox where they can search for a client.

As the client base increased, this turns out to be exceptionally slow. Around 1 minute for a operation that happens around 100 times each day.

Currently MSSQL management studio says the query select id, name, birth_date from client takes 41 seconds to execute (around 130000 rows).

Is there any suggestions on how to improve this time? Indexes, ORMs or direct sql queries on code?

Currently I'm using framework 3.5 and LinqToSql

Answer 1

If your query is actually SELECT id, name, birth_date from client (ie, no where clause) there is very little that you'll be able to do to speed that up short of new hardware. SQL Server will have to do a table scan to get all of the data. Even a covering index means that it will have to scan an index just as big as the table.

What you need to ask yourself is: is a list of 130000 clients really useful for your users? I anybody really going to scroll through to the 75613th entry in a list to find the user that they want? The answer is probably not. I would go with the search option only. At least then you can add indices that make sense for those queries.

If you absolutely do need the entire list, try loading it lazily in chunks. Start with the first 500 records and then add more records as the user moves the scroll bar. That way the initial load time is reduced and the user will only load the data that is necessary.

Answer 2

Why do you need the list of all the clients? Couldn't you just have the search TextBox that you describe and handle the search query on the server side. There you set a cap on the maximum number of returned rows for an individual client search (eg max 500 matches).

Alternatively, some efficiency gains may be achived by caching the client data list on the web server

Answer 3

Indexing should not help, based on your query. You could use a view which caches the sorted query (assuming you're not ordering by the id?), but given SQL Server's baked-in query cache for adhoc queries you're probably not going to see much gain there either. The ORM does add some overhead, but there are several tutorials out there for cutting the cost of that (eg http://www.sidarok.com/web/blog/content/2008/05/02/10-tips-to-improve-your-linq-to-sql-application-performance.html ). Main points there that apply to you are to use compiled queries wherever possible, and turn off optimistic concurrency for read-only data.

An even bigger performance gain could be realized by having your clients not hit the db directly. If you add a service layer in there (not necessarily a web service, but it could be) then the service class or application could put some smart caching in place, which would help by an order of magnitude for read-only queries like this.

Answer 4

Go in to SQL Server, do a new query. In the Query menu click the "Include Client Statistics".

Run the query just as you would from code. It will display the results and also a tab next to the result called "Client Statistics"

Click that and look at the time in the "Wait time on server replies" This is in ms, and it's the time the server was actually executing.

I just ran this query:

select  firstname, lastname from leads

It took 3ms on the server to fetch 301,000 records.

The "Total Execution Time" was something like 483ms, which includes the time for SSMS to actually get the data and process it. My query took something like 2.5-3s to run in SSMS and the remaining time (2500ms or so) was actually for SSMS to paint the results etc.)

My guess is, the 41 seconds is probably not being spent on the SQL server, as 130,000 records really isn't that much. Your 41 seconds is probably largely being spent by everything after the SQL server returns the results.

If you find out SQL Server is taking a long time to execute, in the query menu turn on "Include Actual Execution Plan" Rerun your query. A new tab appears called "Execution Plan" this tab will show you what SQL server is doing when you do a select on this table as well as a percentage of where it spends all of it's time. In my case it spent 100% of the time in a "Clustered Index Scan" of PK_Leads

Edited to include more stats

Answer 5

In general:

1. Find out what takes so much time, executing the query or retrieving the results
2. If its the query execution, the query plan will tell you which indexes are missing, just press the display query plan button in the SSMS and you will get hints on which indexes you should create to increase performance
3. If its the retrieval of the values, there is not much you can do about it besides upgrading hardware (ram, disk, network etc.)

But:
In your case it looks like the query is a full table scan , which is never good for performance, check if you really need to retrieve all this data at once.
Since there are no clauses what so ever its unlikely that its the query execution that is the problem. Meaning additional indexes will not help.

You will need to change the way the application access the data. Instead of loading all clients into memory and then search from them in memory you will need to pass on the search term to the database query.

LinqToSql enable you to use different features for searching values, here is a blog describing most of them: http://davidhayden.com/blog/dave/archive/2007/11/23/LINQToSQLLIKEOperatorGeneratingLIKESQLServer.aspx