[英]protobuf-csharp-port - streaming records from a file a bit like an axis function in LINQ-to-XML
I have built the standard address book tutorial that comes with protobuf-csharp-port and my code is as follows: 我建立了protobuf-csharp-port随附的标准地址簿教程,我的代码如下:
class Program
{
static void Main(string[] args)
{
CreateData();
ShowData();
}
private static void CreateData()
{
AddressBook.Builder abb = new AddressBook.Builder();
for (int i = 0; i < 2000000; i++)
{
Person.Builder pb = new Person.Builder();
pb.Id = i;
pb.Email = "mytest@thisisatest.com";
pb.Name = "John" + i;
abb.AddPerson(pb.Build());
}
var ab = abb.Build();
var fs = File.Create("c:\\testaddressbook.bin");
ab.WriteTo(fs);
fs.Close();
fs.Dispose();
}
private static void ShowData()
{
var fs = File.Open("c:\\testaddressbook.bin", FileMode.Open, FileAccess.Read, FileShare.Read);
CodedInputStream cis = CodedInputStream.CreateInstance(fs);
cis.SetSizeLimit(Int32.MaxValue);
AddressBook ab = AddressBook.ParseFrom(cis);
Console.WriteLine("Person count: {0}", ab.PersonCount);
for (int i = 0; i < ab.PersonCount; i++)
Console.WriteLine("Name: " + ab.GetPerson(i).Name);
Console.WriteLine("Person count: {0}", ab.PersonCount);
fs.Close();
}
}
On writing the data it takes up 300 MB of RAM for 2m records. 写入数据时,它需要300 MB的RAM用于2m条记录。 On reading it takes up about 415 MB of RAM.
读取时会占用大约415 MB的RAM。
In the XML world, I would stream the elements using an axis function. 在XML世界中,我将使用轴函数流式传输元素。 Is it possible to stream the records inside the address book model object?
是否可以在通讯簿模型对象中流式传输记录? Or maybe there's another way to implement this for more efficient memory-use?
也许还有另一种方法可以实现此目的以更有效地使用内存?
thanks 谢谢
Yes, you can stream both reading and writing. 是的,您可以同时阅读和写作。
There's a version supported by the official Java API and also in my C# API, using WriteDelimitedTo
/ ParseDelimitedFrom
. 官方Java API和我的C#API支持一个版本,使用
WriteDelimitedTo
/ ParseDelimitedFrom
。
Alternatively, you can use MessageStreamWriter
and MessageStreamIterator
, which I introduced into my API before the delimited API came along. 另外,您可以使用
MessageStreamWriter
和MessageStreamIterator
,它们是在分隔API出现之前在API中引入的。
I can't comment on that implementation, but in protobuf-net streaming is fully possible. 我无法对此实现发表评论,但是在protobuf-net中完全可以进行流传输。 If all the objects you want to stream are first-level children of the root object, then you can simply iterate over the outer sequence;
如果您要流式传输的所有对象都是根对象的第一级子代,则可以简单地遍历外部序列; using
Serializer.DeserializeItems<T>
if they are all the same type, or Serializer.NonGeneric.TryDeaerializeWithLengthPrefix
if there are different types of objects involved. 使用
Serializer.DeserializeItems<T>
如果它们都是相同类型的,或Serializer.NonGeneric.TryDeaerializeWithLengthPrefix
如果存在不同类型的涉及对象。
If the item you want to treat as a stream is in the middle of the tree, you can provide an alternative receiving model; 如果要作为流处理的项目位于树的中间 ,则可以提供备用的接收模型。 by just implementing IEnumerable and Add() on a fake collection, it can push data through any API you want (event-based, for example - SAX like).
通过仅在假集合上实现IEnumerable和Add(),它就可以通过所需的任何API推送数据(例如,基于事件的-类似SAX)。
I should also note that you can serialize streaming data in exactly the same ways. 我还应该注意,您可以采用完全相同的方式序列化流数据。 It is not required to have a complete object model at any point.
在任何时候都不需要具有完整的对象模型。
If you want a more complete example, let me know. 如果您需要更完整的示例,请告诉我。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.