TStream to TStringList fails for unicode Delphi

Question

As shown in Test2 of the following code, a TStringList is converted to a TStream and then the TStream is converted back to a TStringList. However, in Delphi 7 Test2 gives the same as Test1 . In unicode Delphi, Test2 does not give correct result. Could you help to suggest what is wrong ?

program Project1;

{$APPTYPE CONSOLE}
{$R *.res}

uses
  Classes, SysUtils;

// http://stackoverflow.com/questions/732666
// Converting TMemoryStream to String in Delphi 2009
function MemoryStreamToString(M: TMemoryStream): string;
begin
  SetString(Result, PChar(M.Memory), M.Size div SizeOf(Char));
end;

procedure Test1;
var
  SrcList: TStrings;
  S: String;
  AStream: TStream;
begin
  SrcList := TStringList.Create;
  try
    with SrcList do
    begin
      Add('aa');
      Add('bb');
      Add('cc');
    end;

    S := SrcList.Text;

    AStream := TMemoryStream.Create;
    try
      // AStream.Write(S[1], Length(S));
      // AStream.Write(S[1], Length(S) * SizeOf(Char));
      AStream.Write(Pointer(S)^, Length(S) * SizeOf(Char));

      WriteLn(MemoryStreamToString(TMemoryStream(AStream)));
    finally
      AStream.Free;
    end;
  finally
    SrcList.Free;
  end;
end;

procedure Test2;
var
  SrcList: TStrings;
  S: String;
  AStream: TStream;
  DestList: TStringList;
  I: Integer;
begin
  SrcList := TStringList.Create;
  try
    with SrcList do
    begin
      Add('aa');
      Add('bb');
      Add('cc');
    end;

    S := SrcList.Text;

    AStream := TMemoryStream.Create;
    try
      // AStream.Write(S[1], Length(S));
      // AStream.Write(S[1], Length(S) * SizeOf(Char));
      AStream.Write(Pointer(S)^, Length(S) * SizeOf(Char));

      DestList := TStringList.Create;
      try
        AStream.Position := 0;
        DestList.LoadFromStream(AStream);

        WriteLn(DestList.Text);
      finally
        DestList.Free;
      end;
    finally
      AStream.Free;
    end;
  finally
    SrcList.Free;
  end;
end;

begin
  try
    Test1;
    Test2;
  except
    on E: Exception do
      WriteLn(E.ClassName, ': ', E.Message);
  end;
  Readln;
end.

Answer 1

Test1 writes the raw String data as-is into the TMemoryStream and then reads it back as-is into a String , thus everything matches correctly in all versions of Delphi. Test2 works in Delphi 2007 and earlier for a similar reason.

Test2 fails in Delphi 2009 and later because you are not taking into account that TStrings.LoadFrom...() (and TStrings.SaveTo...() ) is TEncoding -aware in those versions of Delphi. You are writing UTF-16 encoded data into the TMemoryStream without a BOM in front, and then you are not telling LoadFromStream() that the stream is UTF-16 encoded. It tries to locate a BOM, and when it does not find one, it loads the stream using TEncoding.Default (aka 8bit Ansi) instead (for backwards compatibility with legacy code).

So, in this example, you need to specify the proper encoding of the stream data when loading it in Delphi 2009 and later:

DestList.LoadFromStream(AStream, TEncoding.Unicode);