Delphi and Indy TIdFTP: Copy all files from one folder on the server to another

Question

I'm using TIdFTP (Indy 10.6) for a client application and I need to be able to copy all files from one folder on the server to another. Can this be done?

I know how to rename or move a file, we can use TIdFTP.Rename(Src, Dst) . How about the copy? Would I need to use Get() and Put() with a new path / name, knowing that the number of files in the server can exceed 500,000 files.

In our company, we have some files whose size exceeds 1.5 GB. By using my code, it consumes a lot of memory and the file is not copied from one directory to another: in less code, the source directory is named "Fichiers" and the destination directory is named "Sauvegardes".

Here is my code:

var
  S , directory  : String;
  I: Integer;
  FichierFTP : TMemoryStream;
begin

  IdFTP1.Passive := True;

  idftp1.ChangeDir('/Fichiers/'); 
  IdFTP1.List();

  if IdFTP1.DirectoryListing.Count > 0 then begin

    IdFTP1.List();

    for I := 0 to IdFTP1.DirectoryListing.Count-1 do begin

      with IdFTP1.DirectoryListing.Items[I] do begin

        if ItemType = ditFile then begin

          FichierFTP := TMemoryStream.Create;

          S := FileName;

          idftp1.Get( FileName , FichierFTP , false );

          Application.ProcessMessages
          idftp1.ChangeDir('/Sauvegardes/' ); 
          idftp1.Put(FichierFTP , S );

          Application.ProcessMessages;
          FichierFTP.Free;  
        end;
      end;
    end;
    IdFTP1.Disconnect;
  end;

Does anyone have any experience with this? How can I change my code to resolve this problem?

Answer 1

There are no provisions in the FTP protocol, and thus no methods in TIdFTP , to copy/move multiple files at a time. Only to copy/move individual files one at a time.

Moving a file from one FTP folder to another is easy, that can be done with the TIdFTP.Rename() method. However, copying a file typically requires issuing separate commands to download the file locally first and then re-upload it to the new path.

Some FTP servers support custom commands for copying files, so that you do not need to download/upload them locally. For example, ProFTPD's mod_copy module implements SITE CPFR/CPTO commands for this purpose. If your FTP server supports such commands, you can use the TIdFTP.Site() method, eg:

Item := IdFTP1.DirectoryListing[I];
if Item.ItemType = ditFile then
begin
  try
    IdFTP1.Site('CPFR ' + Item.FileName);
    IdFTP1.Site('CPTO /Sauvegardes/' + Item.FileName);
  except
    // fallback to another transfer option, see further below...
  end;
end;

If that does not work, another possibility to avoid having to copy each file locally is to use a site-to-site transfer between 2 separate TIdFTP connections to the same FTP server. If the server allows this, you can use the TIdFTP.SiteToSiteUpload() and TIdFTP.SiteToSiteDownload() methods to make the server transfer files to itself, eg:

IdFTP2.Connect;
...
Item := IdFTP1.DirectoryListing[I];
if Item.ItemType = ditFile then
begin
  try
    IdFTP1.SiteToSiteUpload(IdFTP2, Item.FileName, '/Sauvegardes/' + Item.FileName);
  except
    try
      IdFTP2.SiteToSiteDownload(IdFTP1, Item.FileName, '/Sauvegardes/' + Item.FileName);
    except
      // fallback to another transfer option, see further below...
    end;
  end;
end;
...
IdFTP2.Disconnect;

But, if using such commands is simply not an option, then you will have to resort to downloading each file locally and then re-uploading it. When copying a large file in this manner, you should use TFileStream (or similar) instead of TMemoryStream . Do not store large files in memory. Not only do you risk a memory error if the memory manager can't allocate enough memory to hold the entire file, but once that memory has been allocated and freed, the memory manager will hold on to it for later reuse, it does not get returned back to the OS. This is why you end up with such high memory usage when you transfer large files, even after all transfers are finished.

If you really want to use a TMemoryStream , use it for smaller files only. You can check each file's size on the server (either via TIdFTPListItem.Size if available, otherwise via TIdFTP.Size() ) before downloading the file, and then choose an appropriate TStream -derived class to use for that transfer, eg:

const
  MaxMemoryFileSize: Int64 = ...; // for you to choose...
var
  ...
  FichierFTP : TStream;
  LocalFileName: string;
  RemoteFileSize: Int64;

Item := IdFTP1.DirectoryListing[I];
if Item.ItemType = ditFile then
begin
  LocalFileName := '';
  if Item.SizeAvail then
    RemoteFileSize := Item.Size
  else
    RemoteFileSize := IdFTP1.Size(Item.FileName);
  if (RemoteFileSize >= 0) and (RemoteFileSize <= MaxMemoryFileSize) then
  begin
    FichierFTP := TMemoryStream.Create;
  end else
  begin
    LocalFileName := MakeTempFilename;
    FichierFTP := TFileStream.Create(LocalFileName, fmCreate);
  end;
  try
    IdFTP1.Get(Item.FileName, FichierFTP, false);
    IdFTP1.Put(FichierFTP, '/Sauvegardes/' + Item.FileName, False, 0);
  finally
    FichierFTP.Free;
    if LocalFileName <> '' then
      DeleteFile(LocalFileName);
  end;
end;

There are other optimizations you can make to this, for instance creating a single TMemoryStream with a pre-sized Capacity and then reuse it for multiple transfers that will not exceed that Capacity .

---

So, putting this all together, you could end up with something like the following:

var
  I: Integer;
  Item: TIdFTPListItem;
  SourceFile, DestFile: string;
  IdFTP2: TIdFTP;
  CanAttemptRemoteCopy: Boolean;
  CanAttemptSiteToSite: Boolean;

  function CopyFileRemotely: Boolean;
  begin
    Result := False;
    if CanAttemptRemoteCopy then
    begin
      try
        IdFTP1.Site('CPFR ' + SourceFile);
        IdFTP1.Site('CPTO ' + DestFile);
      except
        CanAttemptRemoteCopy := False;
        Exit;
      end;
      Result := True;
    end;
  end;

  function CopyFileSiteToSite: Boolean;
  begin
    Result := False;
    if CanAttemptSiteToSite then
    begin
      try
        if IdFTP2 = nil then
        begin
          IdFTP2 := TIdFTP.Create(nil);
          IdFTP.Host := IdFTP1.Host;
          IdFTP.Port := IdFTP1.Port;
          IdFTP.UserName := IdFTP1.UserName;
          IdFTP.Password := IdFTP1.Password;
          // copy other properties as needed...
          IdFTP2.Connect;
        end;
        try
          IdFTP1.SiteToSiteUpload(IdFTP2, SourceFile, DestFile);
        except
          IdFTP2.SiteToSiteDownload(IdFTP1, SourceFile, DestFile);
        end;
      except
        CanAttemptSiteToSite := False;
        Exit;
      end;
      Result := True;
    end;
  end;

  function CopyFileManually: Boolean;
  const
    MaxMemoryFileSize: Int64 = ...;
  var
    FichierFTP: TStream;
    LocalFileName: String;
    RemoteFileSize: Int64;
  begin
    Result := False;
    try
      if Item.SizeAvail then
        RemoteFileSize := Item.Size
      else
        RemoteFileSize := IdFTP1.Size(SourceFile);
      if (RemoteFileSize >= 0) and (RemoteFileSize <= MaxMemoryFileSize) then
      begin
        LocalFileName := '';
        FichierFTP := TMemoryStream.Create;
      end else
      begin
        LocalFileName := MakeTempFilename;
        FichierFTP := TFileStream.Create(LocalFileName, fmCreate);
      end;
      try
        IdFTP1.Get(SourceFile, FichierFTP, false);
        IdFTP1.Put(FichierFTP, DestFile, False, 0);
      finally
        FichierFTP.Free;
        if LocalFileName <> '' then
          DeleteFile(LocalFileName);
      end;
    except
      Exit;
    end;
    Result := True;
  end;

begin
  CanAttemptRemoteCopy := True;
  CanAttemptSiteToSite := True;

  IdFTP2 := nil;
  try
    IdFTP1.Passive := True;

    IdFTP1.ChangeDir('/Fichiers/'); 
    IdFTP1.List;

    for I := 0 to IdFTP1.DirectoryListing.Count-1 do
    begin
      Item := IdFTP1.DirectoryListing[I];

      if Item.ItemType = ditFile then
      begin
        SourceFile := Item.FileName;
        DestFile := '/Sauvegardes/' + Item.FileName;

        if CopyFileRemotely then
          Continue;

        if CopyFileSiteToSite then
          Continue;

        if CopyFileManually then
          Continue;

        // failed to copy file! Do something...
      end;
    end;
  finally
    IdFTP2.Free;
  end;

  IdFTP1.Disconnect;
end;