简体繁体 English

读取大文件的最佳方法

[英]Best approach to reading large files

原文 2018-03-05 08:06:05 5 2 c#/ xml/ io

I'm currently working on a program that reads writes a XML file. 我当前正在开发一个程序，该程序会读取并写入XML文件。 While this is a simple task, i'm concerned about future issues. 尽管这是一项简单的任务，但我担心将来的问题。

My code reads the streamed data from the XML, and checks every element <x> until an element that matches a criteria is founds, this works quite fast, since the file currently has about 100 <x> elements, but when more elements are added this task will be much slower, specially if the element that matches the criteria is the last one in avery large file. 我的代码从XML读取流数据，并检查每个元素<x>直到找到一个符合条件的元素为止，此方法的运行速度非常快，因为该文件当前具有大约100个<x>元素，但是当添加了更多元素时这项任务将非常慢，特别是如果与标准匹配的元素是每个大文件中的最后一个元素。

What approach should I take to minimize the impact of this? 我应该采取什么方法来最大程度地减少这种影响？ I was thinking about spliting files in smaller ones (containing up to 1000 elements each) and reading from various of those files at the same time. 我当时正在考虑将文件拆分为较小的文件（每个文件最多包含1000个元素），并同时读取其中的各种文件。 Is this a proper approach to this? 这是正确的方法吗？

I'm coding in C#, in case it's relevant for a language-specific approach. 我正在用C＃进行编码，以防它与特定于语言的方法有关。

2 个解决方案

You should use one of the available XML APIs of .Net. 您应该使用.Net的可用XML API之一。 Which one depends on the size of the XML files. 哪一个取决于XML文件的大小。 In this question there is a discussion between XDocument (Linq-to-Xml) and XmlReader . 在这个问题中，在XDocument （Linq-to-Xml）和XmlReader之间进行了讨论。 To summarize: If your file fits in memory, then use XDocument . 总结一下：如果文件适合内存，请使用XDocument 。 If not then use XmlReader . 如果没有，请使用XmlReader 。

This sounds like a batch process in your case. 在您的情况下，这听起来像一个批处理过程。 Maybe this link: https://www.codeproject.com/Articles/1155341/Batch-Processing-Patterns-with-Taskling will help you. 也许此链接： https : //www.codeproject.com/Articles/1155341/Batch-Processing-Patterns-with-Taskling将为您提供帮助。 I never did this in C#, but in Java, and it's a good way to resolve this kind of tasks. 我从来没有在C＃中这样做，但是在Java中，这是解决此类任务的好方法。 Hope it will help you. 希望对您有帮助。