Ways to speed up queries SQL Server 2008 R2 without SqlDataSource object

Question

I'm trying to build a product catalog application in ASP.NET and C# that will allow a user to select product attributes from a series of drop-down menus, with a list of relevant products appearing in a gridview.

On page load, the options for each of the drop-downs are queried from the database, as well as the entire product catalog for the gridview. Currently this catalog stands at over 6000 items, but we're looking at perhaps five or six times that when the application goes live.

The query that pulls this catalog runs in less than a second when executed in SQL Server Management Studio, but takes upwards of ten seconds to render on the web page. We've refined the query as much as we know how: pulling only the columns that will show in our gridview (as opposed to saying select * from ... ) and adding the with (nolock) command to the query to pull data without waiting for updates, but it's still too slow.

I've looked into SqlCacheDependency , but all the directions I can find assume I'm using a SqlDataSource object. I can't do this because every time the user makes a selection from the menu, a new query is constructed and sent to the database to refine the list of displayed products.

I'm out of my depth here, so I'm hoping someone can offer some insight. Please let me know if you need further information, and I'll update as I can.

EDIT: FYI, paging is not an option here. The people I'm building this for are standing firm on that point. The best I can do is wrap the gridview in a div with overflow: auto set in the CSS.

The tables I'm dealing with aren't going to update more than once every few months, if that; is there any way to cache this information client-side and work with it that way?

Answer 1

Most of your solution will come in a few forms (none of which have to do with a Gridview):

1. Good indexes. Create good indexes for the tables that pull this data; good indexes are defined as:

   • Indexes that store as little information as actually needed to display the product. The smaller the amount of data stored, the greater amount of data can be stored per 8K page in SQL Server.
   • Covering indexes: Your SQL Query should match exactly what you need (not SELECT * ) and your index should be built to cover that query (hence why it's called a 'covering index')

2. Good table structure: this goes along with the index. The fewer joins needed to pull the information, the faster you can pull it.

3. Paging. You shouldn't ever pull all 6000+ objects at once -- what user can view 6000 objects at once? Even if a theoretical superhuman could process that much data; that's never going to be your median usecase. Pull 50 or so at a time (if you really even need that many) or structure your site such that you're always pulling what's relevant to the user, instead of everything (keep in mind this is not a trivial problem to solve)

The beautiful part of paging is that your clients don't even need to know you've implemented paging. One such technique is called " Infinite Scrolling ". With it, you can go ahead and fetch the next N rows while the customer is scrolling to them.

Answer 2

If, as you're saying paging really is not an option (although I really doubt it ; please explain why you think it is, and I'm pretty sure someone will find a solution), there's really no way to speed up this kind of operation.

As you noticed, it's not the query that's taking long, it's the data transfer. Copying the data from one memory space (sql) to another (your application) is not that fast, and displaying this data is orders of magnitude slower.

Edit: why are your clients "firm on that point" ? Why do they think it's not possible otherwise ? Why do they think it's the best solution ?

Answer 3

There are many options to show a big largeset of data on a grid but third parties software. Try to use jquery/javascript grids with ajax calls. It will help you to render on client a large amount of rows. Even you can use the cache to not query many times the database. Those are a good grids that will help your to show thousands of rows on a web browser:

1. http://www.trirand.com/blog/
2. https://github.com/mleibman/SlickGrid
3. http://demos.telerik.com/aspnet-ajax/grid/examples/overview/defaultcs.aspx
4. http://w2ui.com/web/blog/7/JavaScript-Grid-with-One-Million-Records

I Hope it helps.

Answer 4

You can load all the rows into a Datatable on the client using a Background thread when the application (Web page) starts. Then only use the Datatable to populate your Grids etc....So you do not have to hit SQL again until you need to read / write different data. (All the other answers cover the other options)