Indy TIdFTP "Out of Memory" error when downloading a large file

Question

I am using TIdFTP (Indy 10.6) to upload a large file that exceeds 1GB.

When I want to download that file, I get an error message saying "Out of Memory".

Here is my code:

var
  FTPfile: TMemoryStream;
begin
  ...
  FTPFile := TMemoryStream.Create;
  ...
  TRY
    IdFTP1.Get ('Myfile.pdf', FTPfile, false);
    FTPfile.Position := 0;
  EXCEPT
    On E: Exception do
      ShowMessage (E.Message);
  END;

I am using a 32bit version of Windows.

Is there a solution to work around this problem?

Answer 1

You are trying to download a 1GB file into a TMemoryStream . Aside from just being a bad idea in general, the main problem with doing this is that the FTP protocol does not report the size of a file that is being transferred, so TIdFTP.Get() can't pre-allocate the TMemoryStream 's internal buffer up front. As such, as the file downloads, the TMemoryStream.Capacity will have to grow many times, and each time it will allocate a completely new buffer, copy the existing data into it, and free the old buffer. This means that during a growth operation, there is a small window of time where 2 buffers exist in memory. So you are actually using much more memory than you think. And storing a 1GB file in memory is really pushing the limit of how much memory a 32bit process can even allocate.

Eventually, the TMemoryStream will grow so large that the system just can't allocate that 2nd buffer anymore, thus failing with the "Out of Memory" error.

You could try using TIdFTP.Size() or TIdFTP.List() to determine the remote file's size ahead of time, set the TMemoryStream.Capacity accordingly, and then call TIdFTP.Get() . This way, there is only 1 memory allocation performed.

Also, make sure you Free() the TMemoryStream when you are done using it, or else you will leak whatever memory it was able to allocate.

But really, you should download the file to disk instead. You can use a TFileStream for that, but TIdFTP.Get() has an overload that takes a local file path as the destination instead of a TStream .

If you need to access the file's data in memory, you can read/memory-map it as needed after the download is complete.