extra spaces with string to buffer void type conversion implicit in Filestream.WriteBuffer method

Question

Haven't needed to post here for a while, but I have a problem implementing filestreams. When writing a string to filestream, the resultnig text file has extra spaces inserted between each character

So when running this method:

Function TDBImportStructures.SaveIVDataToFile(const AMeasurementType: integer;
  IVDataRecordList: TIV; ExportFileName, LogFileName: String;
  var ProgressInfo: TProgressInfo): Boolean; // AM
var
  TempString: unicodestring;
  ExportLogfile, OutputFile: TFileStream;
begin
  ExportLogfile := TFileStream.Create(LogFileName, fmCreate);
  TempString :=
    'FileUploadTimestamp, Filename, MeasurementTimestamp, SerialNumber, DeviceID, PVInstallID,'
    + #13#10;
  ExportLogfile.WriteBuffer(TempString[1], Length(TempString) * SizeOf(Char));
  ExportLogfile.Free;

  OutputFile := TFileStream.Create(ExportFileName, fmCreate);
  TempString :=
    'measurementdatetime,closestfiveseconddatetime,closesttenminutedatetime,deviceid,'
    + 'measuredmoduletemperature,moduletemperature,isc,voc,ff,impp,vmpp,iscslope,vocslope,'
    + 'pvinstallid,numivpoints,errorcode' + #13#10;
  OutputFile.WriteBuffer(TempString[1], Length(TempString) * SizeOf(Char));
  OutputFile.Free;
end;

(which is a stripped down test method, writing headers only). The resulting csv file for the 'OutPutFile' reads

'measuredmoduletempera ture, etcetera when viewed in wordpad, but not in excel, notepad, etc. I'm guessing its the SizeOf(Char) statement which is wrong in a unicode context, but I'm not sure what would be the correct thing to insert here. The 'ExportLogfile' seems to work ok but not the 'OutPutFile'

From what I've read elsewhere it is the writing in unicode which is the problem & not WordPad, see http://social.msdn.microsoft.com/Forums/en-US/7e040fd1-f399-4fb1-b700-9e7cc6117cc4/unicode-to-files-and-console-vs-notepad-wordpad-word-etc?forum=vcgeneral

Any suggestions folks? many thanks, Brian

Answer 1

You are writing 16 bit UTF-16 encoded characters. And then viewing the text as if it were ANSI encoded text. This mismatch explains the behaviour. In fact you don't have extra spaces, those are zero bytes, interpreted as null characters.

You need to decide which encoding you wish to use. Which programs will read the file? Which text encoding are they expecting? Few programs that read csv files understand UTF-16.

A quick fix would be to switch to using AnsiString which would result in 8 bit text. But would not support international text. Do you need to support international text? Then perhaps you need UTF-8. Again you could perform a quick fix using Utf8String , but I think you should look deeper.

It's odd that you handle the text to binary conversion. It would be much simpler to use TStringList , calling Add to add lines, and then specify an encoding when saving the file.

List.Add(...);
List.Add(...);
// etc. 
List.SaveToFile(FileName, TEncoding.UTF8);

A perhaps more elegant approach would be to use the TStreamWriter class. Supply an output stream (or filename) and an encoding when creating the object. And then call Write or WriteLine to add text.

Writer := TStreamWriter.Create(FileName, TEncoding.UTF8);
try
  Writer.WriteLine(...);
  // etc. 
finally
  Writer.Free;
end;

I've assumed UTF-8 here but you can easily specify a different encoding.