TClientDataset to edit a table with 100k+ records

Question

A client wants to build a worksheet-like application to show data from a database (presumably on a TDbGrid or similar), allowing free search and edition of all cells, as you would do in a worksheet. Underlying table will have more than 100k rows.

The problem with using TClientDataset, is that it tends to load all data into memory, violating user requisites, that are these 3:

1. User will be able navigate from first to last record at any moment using scroll bar, keyboard, or a search filter (note that TClientDataset will load all records if you go to last record, AFAIK...).

2. Connection will be through external VPN / internet (possibly slow), so only the actual visible records on screen should be loaded. Never all .

3. Editions must be kept inside a transaction , so they can be committed or rollbacked at the end, and reconciling if necessary.

Is it possible to accomplish this 3 points using TClientDataset? If not, what are the alternatives?

Answer 1

I'm answering just by your last line regarding alternatives, I can add some suggestions:

1- You can use some creativity, provide pagination and fetch let's say 100 rows per page using a thread which is equipped with a nice progress bar in the UI. in this method you must manage search and filters by some smart queries, reloading data sometimes, etc...

2- Use third party components that optimized for this purpose like SDAC + EhLib Dbgrid. SDAC is a dataset that can be useful for cache updates and EhDBGrid has a MemTable component inside it which is very powerful, free search, fuzzy match or approximate search work nicely, possible to revert, undo and redo, etc...