How do I process large amount of data in asp.net

Question

I have a web project in asp.net/C#/Mysql where the data can be up to some 30,000 rows of data to process. Its a reporting tool, and I have to show statistics like counts and sum at several levels.

I want to know which would be the better way to go around this.

I can filter my data to limited columns though which I can query.

1. Now, Is is a good way to get the data (all rows) to my application on load and whenever user queries I can filter that data and do my calculations in the code and show my statistics.

2. or I can have a stored procedure do all my calculations and every time user queries I can call the stored procedure and get my statistics.

Thanks

Answer 1

Databases are optimized to do this kind of data manipulation. And since you reduce network load as well i would vote for the second option.

Another possibility is to consider some kind of OLAP solution where transactional data is already consolidated into smaller chunks of data which in turn can be easily queried.

Answer 2

I would definetly go with the second option. You are talking about a web application, if You want to get all the data at load, then you must store it somewhere to preserve it during postbacks. If you chose to store it in a session state, you will end up consuming the Web server memory, since you have more than one user accessing your site.

If you store it in view state, then you will end up in a very big client response,which will make your page very slow to load on the client side, and will cause network traffic.

Option 2 is the best, because stored procedures are precompiled, which means they are much better in terms of performance. You will also reduce network traffic.