管理大型数据集

Question

I am developing a web based data analysis system and want to know what the best practices are when it comes to managing large datasets. I'm housing the data in a MySQL DB and using php.

For example if I wanted to be able to filter a data table on the fly I am aware of 2 options:

1. Query the db once for all data and use javascript to filter data on the client side or
2. Execute a new query to update the data each time a filter changes

It seems to me that option 1 is the better/quicker option but is this a bad idea when the dataset get very big (ie >100k rows)

Maybe there are other better ways to tackle this but as of now I'm not aware of any so any suggestions would be much appreciated!

Thanks

Answer 1

The best is to query the db only for your needs.

Then you load via ajax and query the other parts that you need when you need it.

Rendering 100k rows on the client side will take too much time ;)