slow mysql database restore with multiple threads (delphi)

Question

I need to restore a lot of mysql database backups and I've been trying to speed up by using multiple threads (in Delphi), each with their own connection. When I'm using MODE_SCRIPT, I can only process around 1 file per second (fps), with the CPU/DISK/MEMORY not stressed at all

When I'm using MODE_CMD, I can get as high as 12+fps with the CPU up to 100% on all cores.

It looks like when using TClientDataSet or descendants, the script is not using all cores, even when using multiple threads?

Minimal code example:

type
  TWorker = class(TThread)
  private
    FTasks: TThreadStringList;
    FConn: TMyConnection;
    FScript: TMyScript;
    FQ: TMyQuery;
  protected
    procedure Execute; override;
  public
    procedure addTask(const aFn: String);
    constructor create(Suspended: Boolean; const aMyId: LongInt;const aIniDb: TIniDBSettings);
  end;

procedure TWorker.addTask(const aFn: String);
begin
  FTasks.Add(aFn);
end;

constructor TWorker.create(Suspended: Boolean; const aMyId: LongInt; const aIniDb: TIniDBSettings);
begin
  inherited Create(Suspended);
  FTasks := TMTThreadStringList.Create;
  FMyName := 'WORKER__'+IntToStr(aMyId);
end;

procedure TWorker.Execute;
var
  mode: LongInt;
const
    MODE_DOS=1;
    MODE_SCRIPT = 2;
begin  
  FConn := TMyConnection.Create(Nil);
  FConn.Username := aIniDb.iniSDBUsername;
  FConn.Password := aIniDb.iniSDBPass;
  FConn.Database := aIniDb.iniSDBDatabase;
  FConn.Server := aIniDb.iniSDBServer;

  FScript := TMyScript.Create(Nil);
  FScript.Connection := FConn;

    try
        FConn.Connect;
        while not Terminated do begin
          if FTasks.Count > 0 then begin
            tmpFn := FTasks.Strings[0];
            FTasks.Delete(0);

            fMyDbname := 'tmpdb_'+FMyName;

            if(mode=MODE_SCRIPT) then {
                FQ.SQL.Text := 'drop database if exists '+fMyDbname ;
                FQ.Execute;
                FQ.SQL.Text := 'create database '+fMyDbname;
                FQ.Execute;
                FQ.SQL.Text := 'use '+fMyDbname;
                fQ.Execute;
                FScript.SQL.LoadFromFile(tmpFn+'.new');
                FScript.Execute;
            }
            else if(mode=MODE_DOS) then begin
                sCmd := 'cmd.exe /c mysql -u user -h serverip < '+tmpFn;
                GetDosOutput(sCmd,dosOutput);//function using 'CreateProcess()'
            }
            InterlockedIncrement(QDONE);
          end
          else Sleep(15);
        end;
      except on e: Exception do
        MessageBox(0,PWideChar('error'+e.Message),'error',MB_OK);
      end;
    end;

Answer 1

It sounds like you are using MyISAM. That is antiquated, and suffers from "table locks", which inhibits much in the way of parallelism.

The following are irrelevant for MyISAM:

-SET FOREIGN_KEY_CHECKS=0;
-SET autocommit=0;

Some questions that relate to the problem:

• Do you have AUTO_INCREMENT columns?
• Are you inserting into the same table at the same time from different threads? (Problematic with MyISAM and MEMORY, less so with InnoDB.)
• How many UNIQUE keys on each table? ( INSERTs are slowed down by the need to check for dups.)
• Are you using INSERT ? One row at a time? Or batched? (Inserting a batch of 100 rows at a time is about optimal -- 10 times as fast as 1 at a time.)
• Or are you using LOAD DATA ? (Even faster.)
• What is the relationship between a "file" and a "table"? That is, are you loading lots of little files into a table, or each file is one table?
• Does the RAID have striping and/or a Battery Backed Write Cache?
• Is the disk HDD or SSD?
• What is the ping time between the client and server? (You mentioned "network", but gave no indication of proximity.)
• How many tables? Are you creating up to 1.87 tables per second? That is 3 files to write and 1 to read? (Windows is not the greatest at rapid opening of files.) That's about 7 file opens/sec. (Note InnoDB needs only 1 file per table if using innodb_file_per_table=1 .)

Please provide SHOW CREATE TABLE for a couple of the larger tables. Please provide a sample of the SQL statements used.

Wilson's request could also be handy.