Exporting from SQLite to Excel in Delphi

Question

I'm exporting data from SqlLite using TSQLConnection with the following procedure:

  Campos := TStringList.Create;
  SQLiteConnection.Execute('SELECT * FROM tabela', nil, results);
  if not results.IsEmpty then
  begin
    results.GetFieldNames(Campos);

    XLApp1 := createoleobject('excel.application');
    XLApp1.Workbooks.Add(xlWBatWorkSheet);
    Sheet := XLApp1.Workbooks[1].WorkSheets[1];
    linha := 1;
    begin
       pBar.Max := results.RecordCount;
       for coluna := 0 to results.FieldCount - 1 do
           Sheet.Cells[1, coluna + 1]  :=  Campos[coluna];

       while ( not results.Eof ) do
       begin
          linha := linha + 1;
          for coluna := 0 to results.FieldCount - 1 do
              Sheet.Cells[linha, coluna + 1]  :=  results.FieldByName(campos[coluna]).AsString;
          results.next;
          pBar.Position := pBar.Position + 1;
          Application.ProcessMessages;
       end;
   end;

);

It works ok. But it takes too much time to finish. 35 minutes to export a table with 40,000 records and 45 fields on i7 note with SSD.

So my question: is there a chance I could do a little faster?

Answer 1

The easiest way to do it is to collect all of the data into a single variant array, and then pass that over to Excel in a single pass. The vast portion of the time is in writing to Excel, and reducing it to one write operation is much faster.

The code below is adapted from code to transfer data from a TStringGrid to an Excel worksheet. Results is from your original code, so consider that it's been set up exactly as you've done it above where indicated. (The below code is untested as modified, because clearly I don't have your data to test against. It works with the stringgrid, as I've mentioned - I just couldn't test the adaptations. If you run into issues, leave a comment and I'll try to address them.)

With 40K rows of 35 fields, you may need to break it up into blocks (even doing a few hundred rows at a time would be a major performance improvement over doing it one at a time).

var
  xls, wb, Range: OLEVariant;
  arrData: Variant;
  RowCount, ColCount, i, j: Integer;
begin
  // Set up your dataset as Result here, just as in your own code
  // ....

  //create variant array where we'll copy our data
  RowCount := Results.RecordCount;
  ColCount := StringGrid1.FieldCount;
  arrData := VarArrayCreate([1, RowCount, 1, ColCount], varVariant);

  //fill array
  j := 1;
  while not Results.Eof do
  begin
    for i := 1 to ColCount do
      arrData[i, j] := Results.Fields[i].AsString;
    Inc(j);
    Results.Next;
  end;

  //initialize an instance of Excel
  xls := CreateOLEObject('Excel.Application');

  //create workbook
  wb := xls.Workbooks.Add;

  //retrieve a range where data must be placed
  Range := wb.WorkSheets[1].Range[wb.WorkSheets[1].Cells[1, 1],
                                  wb.WorkSheets[1].Cells[RowCount, ColCount]];

  //copy data from allocated variant array
  Range.Value := arrData;

  //show Excel with our data
  xls.Visible := True;
end;