[英]TDictionary save to file
I have a lot of files(about 160 000) and I need to have information about position of individual words in files(fulltext). 我有很多文件(大约16万个),我需要了解各个单词在文件中的位置的信息(全文)。 So I used Dictionary like this: 所以我像这样使用Dictionary:
WordDict : TDictionary<string, TDictionary<string, TIntegerDynArray>>;
Now I know that WORD1 is in FILE1,FILE3 and FILE100 and positions in each file <1,3,5>,<2,8,35> etc. I can fill it, I can use it - its very fast. 现在我知道WORD1在FILE1,FILE3和FILE100中,并且在每个文件<1、3、5>,<2、8、35>等中的位置。我可以填充它,可以使用它-它的速度非常快。 But I don't know how effectively store dictionary to file. 但是我不知道如何有效地将字典存储到文件中。
EDIT: effectively - I mean quickly and small size of file 编辑:有效-我的意思是快速且文件很小
You can use the streaming system of Delphi to write a proprietary stream format. 您可以使用Delphi的流系统来编写专有的流格式。 If size matters (contrary to speed) you can zip the stream. 如果大小很重要(与速度相反),则可以压缩流。 Here is some code: 这是一些代码:
type
TFilePos = TArray<Integer>;
TFileDict = TDictionary<string, TFilePos>;
TWordDict = class (TDictionary<string, TFileDict>)
private
procedure LoadFromStream(stream: TStream);
procedure SaveToStream(stream: TStream);
public
procedure LoadFromZip(const AFileName: string);
procedure LoadFromFile(const AFileName: string);
procedure SaveToZip(const AFileName: string);
procedure SaveToFile(const AFileName: string);
end;
procedure TWordDict.LoadFromZip(const AFileName: string);
var
stream: TStream;
localHeader: TZipHeader;
zipFile: TZipFile;
begin
zipFile := TZipFile.Create;
try
zipFIle.Open(AFIleName, zmRead);
zipFile.Read('worddict', stream, localHeader);
try
LoadFromStream(stream);
finally
stream.Free;
end;
zipFile.Close;
finally
zipFile.Free;
end;
end;
procedure TWordDict.SaveToZip(const AFileName: string);
var
stream: TStream;
zipFile: TZipFile;
begin
stream := TMemoryStream.Create;
try
SaveToStream(stream);
stream.Position := 0;
zipFile := TZipFile.Create;
try
zipFile.Open(AFileName, zmWrite);
zipFile.Add(stream, 'worddict');
zipFile.Close;
finally
zipFile.Free;
end;
finally
stream.Free;
end;
end;
procedure TWordDict.SaveToStream(stream: TStream);
var
posi: System.Generics.Collections.TPair<string, TFilePos>;
i: Integer;
pair: System.Generics.Collections.TPair<string, TFileDict>;
writer: TWriter;
begin
writer := TWriter.Create(stream, 4096);
try
writer.WriteListBegin;
for pair in Self do
begin
writer.WriteString(pair.Key);
writer.WriteListBegin;
for posi in pair.Value do
begin
writer.WriteString(posi.Key);
writer.WriteInteger(Length(posi.Value));
for i in posi.Value do
begin
writer.WriteInteger(i);
end;
end;
writer.WriteListEnd;
end;
writer.WriteListEnd;
finally
writer.Free;
end;
end;
procedure TWordDict.LoadFromStream(stream: TStream);
var
sFiles: TFileDict;
aPosi: TFilePos;
size: Integer;
i: Integer;
sWord: string;
reader: TReader;
sFile: string;
begin
Clear;
reader := TReader.Create(stream, 1024);
try
reader.ReadListBegin;
while not reader.EndOfList do
begin
sWord := reader.ReadString;
sFiles := TFileDict.Create;
reader.ReadListBegin;
while not reader.EndOfList do
begin
sFile := reader.ReadString;
size := reader.ReadInteger;
SetLength(aPosi, size);
for I := 0 to size - 1 do
begin
aPosi[I] := reader.ReadInteger;
end;
sFiles.Add(sFile, Copy(aPosi));
end;
reader.ReadListEnd;
Add(sWord, sFiles);
end;
reader.ReadListEnd;
finally
reader.Free;
end;
end;
procedure TWordDict.LoadFromFile(const AFileName: string);
var
stream: TStream;
begin
stream := TFileStream.Create(AFileName, fmOpenRead);
try
LoadFromStream(stream);
finally
stream.Free;
end;
end;
procedure TWordDict.SaveToFile(const AFileName: string);
var
stream: TStream;
begin
stream := TFileStream.Create(AFileName, fmCreate);
try
SaveToStream(stream);
finally
stream.Free;
end;
end;
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.