Delphi XE Database component that can cache query result

Question

I'm working on a Delphi XE-based desktop program with database, and I have noticed that it will be too big for the program to load an entire table before display it to user per page every time they want to browse for products, but it will be also inefficient for the program to load and reload each page as the user browse like in browser. I want to ask, is there any way for Delphi to cache a query call result, so that for the same query, the database component doesn't query again to the database, but return for the cache instead?

This is for scenario like this:

The program loads the first 20 lines for page 1 from database using query.

SELECT * FROM tbl_product_master LIMIT 0,20;

Then the user hit next page. The program loads the next 20 lines for page 2 from database using query.

SELECT * FROM tbl_product_master LIMIT 21,40;

Then the user hit previous page. The program attempts to load the first 20 lines again.

SELECT * FROM tbl_product_master LIMIT 0,20;

but because the database component knows I have already call this query before, and it already saves the result in the cache, it immediately returns the result from the cache, not firing the query again to the database, thus resulting in faster program.

Is there such a component or configuration exist for Delphi XE? I just need a nudge toward the right direction. Thank you.

Answer 1

(This isn't exactly an answer to your q, but it may provide some usual information and I've included some code which may help inform your implementation).

You didn't mentioned which server back-end you are using, nor what Delphi dataset type you're using to get the data from the server. Others have mentioned TClientDataSet in comments and that is certainly a good place to start if you're interested in client-side cacheing especially considering you can use TDataSetFields to nest Detail table data in the data received by the CDS which is handling the Master.

A thing to bear in mind is that early on (around Delphi 5) the TClientDataSet got a bit of a reputation for having access performance which falls off a cliff once you get into the thousands of rows, though exactly where seems to depend on the number of fields and the record size, amongst other things. The CDS code and Midas.Lib, which it depends upon, have been improved over the years and this has reduced but not entirely eliminated this problem.

Out of interest I wrote a little testbed to demonstrate this possible performance issue and compare its performance with using the TAdoQuery which gets the CDS data from my server to save and load its own data to/from a local disk file. The code I used is shown below and could certainly be improved and made more rigorous.

Broadly it retrieves a certain number of rows from a server table using a TAdoQuery, saves and loads the data from a local disk file, then transfers the data to the CDS using a TDataSetProvider, and saves and loads the CDS data, first with an index on the data's PK and a second time without (because I wanted to see if the CDS could use the PK index to increase the CDS's LoadFromFile performance - it didn't).

Here are the results. Please take them with a large "pinch of salt" because I'm sure others would get different results. My point is, you should investigate this yourself, in your own environment and using your own data and dataset components of choice.

Results

          A      B   C   D     E         F        G      H   I
  Recs  5000    140  63  93   967   0.0001934     952    15  31
  Recs  10000   172 125 156  2574   0.0002574    2355    47  47
  Recs  15000   250 171 219  4508   0.0003005    4477    63  62
  Recs  20000   359 218 297  7082   0.0003541    7129    78  94
  Recs  25000   390 327 343  9985   0.0003994    9968    94 109
  Recs  30000   531 343 421 13401   0.0004467   13572   125 140

A = Number of records

B = Open AdoQuery (milliseconds)

C = Save AdoQuery to file (ms)

D = Load AdoQuery from file (ms)

E = Transfer AdoQuery data to CDS via DataSetProvider with PK index on CDS (ms)

F = E/A, ie transfer time per record

G = Transfer AdoQuery data to CDS via DataSetProvider without PK index on CDS (ms)

H = CDS save to disk file (ms)

I = CDS load from disk file (ms)

Btw, this data had a RecordSize of 241 and 45 fields

Comments/Observations

• The ClientDataSet is quite a lot faster than the AdoQuery in saving & loading data in a local disk. Both sets of times vary roughly linearly with the number of records.

• The AdoQuery's Open times are roughly linear with number of records retrieved.

• The Adoquery-> ClientDataSet transfer time varies a lot worse than linearly with the number of records, which is what I meant by "off a cliff". But as long as you restrict yourself to a few thousand records, the ClientDataSet works just fine. You need to find out what "few" means for your application.

• Increasing the number of records by a factor or 2, from 5000 to 10000 makes the transfer take just over twice the time, while increasing it by a factor of 5, to 30000, multiplies the time by a factor of over 14.

• If I repeat the tests with cbSelectPKOnly.Checked, so that the CDS is only retrieving an integer field, the AdoQuery->ClientDataSet transfer times are quite a bit faster, so you can move the "cliff" by retrieving fewer data columns.

• The code was written in D7 and tested with that (without Andreas Hausladen's admirable "speed fix" for Midas) and XE4. The results are for XE4.

• The host machine included the Sql Server and had all SSD disks, Win7 64-bit.

Code

const
  scPKName = 'ApcsID';
  scSql = 'select top %d %s from btapcs'; // order by apcsid';

function TForm1.GetTestSql(Count : Integer) : String;
var
  S : String;
begin
  if cbSelectPKOnly.Checked then
    S := scPKName
  else
    S := '*';
  Result := Format(scSql, [Count, S]);
end;

procedure TForm1.Button1Click(Sender: TObject);
var
  Count : Integer;
begin
   Count := 5000;
   while Count <= StrToInt(edMaxRecs.Text) do begin
     TestOpenAdoQuery(Count);
     Inc(Count, 5000);
     Application.ProcessMessages;
   end;
end;

procedure TForm1.Log(const Msg : String);
begin
  Memo1.Lines.Add(Msg);
end;

procedure TForm1.TestOpenAdoQuery(Count : Integer);
type
  TDataOperation = (doOpenQuery, doSaveQueryData, doLoadQueryData, doGetCDSData,
    doSaveCDSData, doLoadCDSData);
var
  Msg : String;
  AdoFN,
  CdsFN : String;
  Query : TAdoQuery;
  DSP : TDataSetProvider;
  CDS : TClientDataSet;

  procedure PerformOperation(Op : TDataOperation; var Msg : String);
  var
    T1,
    T2 : Integer;
  begin
    T1 := GetTickCount;
    case Op of
      doOpenQuery : begin
        Query.Sql.Text := GetTestSql(Count);
        Query.Open;
        Msg := Msg + Chr(9) + IntToStr(Query.RecordCount);
      end;
      doSaveQueryData : begin
        Query.SaveToFile(AdoFN);
      end;
      doLoadQueryData : begin
        Query.LoadFromFile(AdoFN);
      end;
      doGetCDSData : begin
        CDS.Open;
      end;
      doSaveCDSData : begin
        CDS.SaveToFile(CdsFN, dfBinary);
      end;
      doLoadCDSData : begin
        CDS.LoadFromFile(CdsFN);
      end;
    end; { case }
    T2 := GetTickCount;
    Msg := Msg + Chr(9) + IntToStr(T2 - T1);
  end;

begin
  //  This proc uses a TAdoConnection1 on the form, but uses a temporary AdoQuery, 
  //  DataSetProvider and TClientDataSet to avoid state being carried over
  //  from one call to the next.

  Screen.Cursor := crSqlWait;
  Update;
  try
    Query := TAdoQuery.Create(Nil);
    Query.CursorType := ctKeySet;
    Query.CursorLocation := clUseClient;
    Query.Connection := AdoConnection1;
    if cbDropConnectionBetweenRuns.Checked then begin
      AdoConnection1.Connected := False;
    end;

    AdoFN := IncludeTrailingPathDelimiter(GetEnvironmentVariable('TEMP'))+ 'ado.dat';
    CdsFN := IncludeTrailingPathDelimiter(GetEnvironmentVariable('TEMP'))+ 'cds.cds';

    Msg := 'Recs ';
    Query.Sql.Text := GetTestSql(Count);
    PerformOperation(doOpenQuery, Msg);

    Query.Connection := Nil;

    PerformOperation(doSaveQueryData, Msg);
    PerformOperation(doLoadQueryData, Msg);

    DSP := TDataSetProvider.Create(Self);
    DSP.Name := 'MyProvider';
    DSP.DataSet := Query;
    CDS := TClientDataSet.Create(Self);

    DSP.DataSet := Query;
    CDS.ProviderName := DSP.Name;

    CDS.IndexFieldNames := scPKName;
    PerformOperation(doGetCDSData, Msg);

    CDS.Close;
    CDS.IndexFieldNames := '';
    PerformOperation(doGetCDSData, Msg);
    PerformOperation(doSaveCDSData, Msg);

    CDS.Close;

    PerformOperation(doLoadCDSData, Msg);

    Log(Msg);

  finally
    Query.Free;
    DSP.Free;
    CDS.Free;
    Screen.Cursor := crDefault;
  end;
end;

Answer 2

There are different ways in Delphi to limit the number of records loaded by a query - it also depends on what database you're using and what data access components.

Some databases will be able to return you only a given number of records until you ask for more, if the Delphi data access library you use also supports that feature and can set it. Some can also cache query results, and avoid to re-execute a query if possible. Already fetched record will be retained.

The TClientDataset + Provider combo offers a similar functionality, independent from the database. It can load data incrementally, and it will cache already fetched rows. It requires a little more code to work, though.

In a native Windows application, you usually don't need the "paging" metaphor web UI uses, because it is far easier to use some control that can scroll up and down while caching data in local memory or disk.

But be aware that:

• If you don't limit the result set at the SQL level, the database may still need to generate the whole result set. The query could run slower, and more server resources are used.
• Some controls may load the whole dataset anyway. For example, Developer Express grids usually load the whole dataset to enable features like sorting, filtering and grouping locally, unless you set them to not do it.
• If you try to get the number of records, and don't do it in a separate SELECT COUNT query or if the db library doesn't return it somehow, the TDataset component may fetch all rows just to get the count.