Processing large XML files using .NET 3.5

Question

What is the "recommended" approach for processing very large XML files in .NET 3.5?

For writing, I want to generate an element at a time then append to a file.

For reading, I would likewise want to read an element at a time (in the same order as written).

I have a few ideas how to do it using strings and File.Append, but does .NET 3.5 provide XML Api's for dealing with arbitrarily large XML files?

Answer 1

Without going into specifics this isn't easy to answer. .NET offers different methods to process XML files:

• XmlDocument creates a DOM, supports XPath queries but loads the entire XML file into memory.
• XElement / XDocument has support for LINQ and also reads the entire XML file into memory.
• XmlReader is a forward-only reader. It does not read the entire file into memory.
• XmlWriter is just like the XmlReader, except for writing

Based on what you say an XmlReader/XmlWriter combination seems like the best approach.

Answer 2

As Dirk said, using an XmlWriter/XmlReader combo sounds like the best approach. It can be very lengthy and if your XML file is fairly complex it gets very unwieldy. I had to do something similar recently with some strict memory constraints. My SO question might come in handy.

But personally, I found this method here on MSDN blogs to be very easy to implement and it neatly handles appending to the end of the XML file without fragments.

Answer 3

Try to make an *.xsd file out of your *.xml. You can than generate *.cs file from *.xsd file. After that load you *.xml file to your object. It should take less memory than whole file.

There is a plugin for VS2010 that gives option to generate *.cs file from *.xsd. It is called XSD2Code. In that plugin you have an option to decorate properties for serialization. For your *.xsd file named Settings you would get Settings.cs. You would than do something like this.

StreamReader str = new StreamReader("SomeFolder\\YourFile.xml");
XmlSerializer xmlSer = new XmlSerializer(typeof(TcpPostavke));
Settings m_settings = (Settings )xmlSer .Deserialize(str);

You can than query your list of objects with Linq.