Huge file support for Delphi 6? (replacement for System module?)

Question

I am running into problems interacting with a huge fixed length record data file. The file is over 14 GB in size. I first noticed a problem when I saw the return value from the System.Filesize() function was far less than the actual number of records in the huge file, given the number of bytes in the file and the length of each record. (System.Filesize returns the number of records in an untyped file given the record size specified during the Reset() call. It does not return the number of bytes in the file). I chalked it up to the return type of System.Filesize() being a Longint instead of an Int64.

I worked around the initial problem by calling GetFileSizeEx() and calculating the number of records myself. Unfortunately, BlockRead() also fails when trying to access records in the file whose offset is deep into the file. I'm guessing that again there are values being used that are overflowing somewhere in the code.

Is there a replacement module out for Delphi 6 there that can handle huge files and is a substitute for the System unit file I/O calls? I'm trying to avoid rolling my own if I can.

Answer 1

You can use the GpHugeFile from Primoz Gabrijelcic . I used this library myself to access larger files (> 2gb) from Delphi 7. Anyway in your case you must consider try to change you app logic and migrate to a Database scheme which is much more efficient which a scheme based in record files.

Answer 2

试试TGpHugeFile 。

Answer 3

It turns out that the internal seek routine used by the System unit also had problems due to the use of low capacity numeric types. I coded up my own call to the Windows SetFilePointerEx() function and all is well. I have provided the source code below in case it might help others. I have included the code I created to get the number of records properly too since you will need both. Everything else works the same.

// Some constants
const
    kernel = 'kernel32.dll';


function SetFilePointerEx(hFile: Integer; distanceToMove: Int64; var newFilePointer: Int64; moveMethod: DWORD): boolean; stdcall; external kernel name 'SetFilePointerEx';


// easyGetFileSize() is a replacement filesize function.  Use it to get the number of bytes in the huge file.  To get the number of records just "div" it by the record size.

function GetFileSizeEx(hFile: THandle; var FileSize: Int64): BOOL; stdcall; external 'kernel32.dll' name 'GetFileSizeEx';


function easyGetFileSize(theFileHandle: THandle): Int64;
begin
    if not GetFileSizeEx(theFileHandle, Result) then
        RaiseLastOSError;
end;

// ---- Replacement seek function.  Use this instead.

procedure mySeek(var f: File; recordSize, recNum: Int64);

var
    offsetInBytes, numBytesRead: Int64;
    pBigInt: ^Int64;
begin
    offsetInBytes := recNum * recordSize;

    pBigInt := nil; // Not interested in receiving a new pointer after seek.

    // Call the Windows seek call since Delphi 6 has problems with huge files.
    if not SetFilePointerEx(TFileRec(f).Handle, offsetInBytes, pBigInt^, FILE_BEGIN) then
        raise Exception.Create(
            '(mySeek) Seek to record number # '
            + IntToStr(recNum)
            + ' failed');
end;

Answer 4

You can't use Pascal I/O with huge files like this, not in any version of Delphi. Your best bet is to use a TFileStream which has no such limitations.