Can I modify a constant in the RTL class System.Classes.TStream and rebuild it at runtime in Delphi XE6?

Question

I am trying to work around a known ugly performance limitation in System.Classes.pas, which has a 1980s era constant buffer limit ($F000) that looks like this:

function TStream.CopyFrom(const Source: TStream; Count: Int64): Int64;
const
  MaxBufSize = $F000;
....

This is causing major performance penalties in our Delphi application. In delphi XE2 through XE5, we were able to modify this and use one of the following approaches:

• I could modify the Delphi sources, and then, by invoking dcc32.exe from a batch file, rebuild the System.Classes.dcu file in the Delphi library folder. I realize this is ugly and I didn't like doing this, but I don't like this ugly performance issue in the RTL either, and our users can not live with the performance headaches it causes.

• I could try to put a modified system.classes.pas file somewhere in my project search path.

Neither of the above approaches is working for me in Delphi XE6, now, thanks probably to some internal compiler changes. The error I get in a minimal command line application that includes System.Contnrs in its uses clause, is this:

[dcc32 Fatal Error] System.Classes.pas(19600): F2051 Unit System.Contnrs was compiled with a different version of System.Classes.TComponent

The sample program to reproduce this problem (assuming you have modified System.Classes.pas and changed the MaxBufSize constant), is shown here:

program consoletestproject;

{$APPTYPE CONSOLE}

{$R *.res}

uses
   System.Contnrs,
   System.SysUtils;

var
  List:System.Contnrs.TObjectList;
begin
   WriteLn('Hello world');
end.

Again, this problem reproduces easily in Delphi XE6, but is not a problem in XE5, or earlier.

What is the recommended practice when you absolutely MUST work around a fundamental RTL or VCL limitation using a modified copy of System.Classes.pas or System.SysUtils.pas or some other very low level unit? (Yes, I know you should NOT do this if you don't have to, don't bother with a lecture.)

Are there a magic set of command line parameters you can use via "dcc32.exe" on the command line, to produce a modified DCU that will link properly with the application example above?

As a secondary question, are there .dcu files for which no source exists that will break when one tries to do this, in which case the answer to all of the above is, "you can't fix this, and if there's a bug in the RTL, you're out of luck"?

One possible workaround is to include "$(BDS)\\source\\rtl\\common" in your project search path (or library path), forcing each broken (needing recompile) DCU to rebuild EACH time, but this seems ugly and wrong.

Answer 1

You can overcome this limitation using a detour, try this sample which uses the Delphi Detours Library

First define the signature of the method to hook

var
 Trampoline_TStreamCopyFrom : function (Self : TStream;const Source: TStream; Count: Int64): Int64 = nil;

then implement the detour

function Detour_TStreamCopyFrom(Self : TStream;const Source: TStream; Count: Int64): Int64;
const
  MaxBufSize = 1024*1024; //use 1 mb now :)
var
  BufSize, N: Integer;
  Buffer: TBytes;
begin
  if Count <= 0 then
  begin
    Source.Position := 0;
    Count := Source.Size;
  end;
  Result := Count;
  if Count > MaxBufSize then BufSize := MaxBufSize else BufSize := Count;
  SetLength(Buffer, BufSize);
  try
    while Count <> 0 do
    begin
      if Count > BufSize then N := BufSize else N := Count;
      Source.ReadBuffer(Buffer, N);
      Self.WriteBuffer(Buffer, N);
      Dec(Count, N);
    end;
  finally
    SetLength(Buffer, 0);
  end;
end;

Finally replace the original function by the trampoline (you can use this code in the initialization part of some unit)

  Trampoline_TStreamCopyFrom     := InterceptCreate(@TStream.CopyFrom,   @Detour_TStreamCopyFrom);

And to release the hook you can use

 if Assigned(Trampoline_TStreamCopyFrom) then
   InterceptRemove(@Trampoline_TStreamCopyFrom);

Answer 2

Update 1: The suggestion below does not work for the Classes unit in XE6. The basic technique is sound and does solve similar problems. But for XE6, at least the Classes unit, it is not immediately obvious how to re-compile it.

This appears to be a fault introduced in XE6 because this technique is meant to work and is officially endorsed by Embarcadero: http://blog.marcocantu.com/blog/2014_august_buffer_overflow_bitmap.html

Update 2:

In XE7, this problem no longer exists. It would appear that whatever was broken in XE6 has been fixed.

---

You need the compiler options to match those used when the unit was compiled by Embarcadero. That's the reason why your implementation section only change fails when it seems like it ought to succeed.

Start a default project and use CTRL + O + O to generate these options. I get

{$A8,B-,C+,D+,E-,F-,G+,H+,I+,J-,K-,L+,M-,N-,O+,P+,Q-,R-,S-,T-,U-,V+,W-,X+,Y+,Z1}

when I do this in XE6.

Put that at the top of your copy of the unit and you should be good to go. You can probably get away with a cut-down subset of these, depending on your host project options. In my code I find that:

{$R-,T-,H+,X+}

suffices.