编辑大型XML文件

Question

I have a 3GB XML file. I need to move nodes as child node of another. Loading large file as XmlDocument is not efficient. I see XmlReader is another approach but not sure exactly how it will work in my scenario and what other classes I should be using to do this.

I need to move all alias node to its related customer>name node.

<customer>
<name><first>Robert</first></name>
<alias>Rob</alias>
</customer>

Answer 1

I don't really understand exactly what transformation you want to perform, but assuming that @dbc's guess is correct, you could do it with a streaming XSLT 3.0 processor like this:

<xsl:transform version="3.0"
 xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:mode streamable="yes" on-no-match="shallow-copy">

<xsl:template match="customer">
  <xsl:apply-templates select="copy-of(.)" mode="local"/>
</xsl:template>

<xsl:mode name="local" streamable="no" on-no-match="shallow-copy"/>

<xsl:template match="name" mode="local">
  <name>
    <xsl:apply-templates mode="local"/>
    <xsl:copy-of select="../alias"/>
  </name>
</xsl:template>

<xsl:template match="alias" mode="local"/>

</xsl:transform>

What's happening here is that everything gets copied in pure streaming mode (tag for tag) until we hit a customer element. When we encounter a customer element we make an in-memory copy of the element and transform it locally using a conventional non-streaming transformation. So the amount of memory needed is just enough to hold the largest customer element.

Answer 2

What you can do is to take the basic logic of streaming an XmlReader to an XmlWriter from Mark Fussell's article Combining the XmlReader and XmlWriter classes for simple streaming transformations to transform your 3GB file into a modified file in which the <alias> nodes have been relocated to the <name> nodes. An example of using such streaming transformations is given in this answer to Automating replacing tables from external files .

Using that answer as a basis, grab the classes XmlReaderExtensions , XmlWriterExtensions , XmlStreamingEditorBase and XmlStreamingEditor from it and subclass XmlStreamingEditor to create CustomerAliasXmlEditor as follows:

class CustomerAliasXmlEditor : XmlStreamingEditor
{
    // Confirm that the <customer> element is not in any namespace.
    static readonly XNamespace customerNamespace = ""; 

    public static void TransformFromTo(string fromFilePath, XmlReaderSettings readerSettings, string toFilePath, XmlWriterSettings writerSettings)
    {
        using (var xmlReader = XmlReader.Create(fromFilePath, readerSettings))
        using (var xmlWriter = XmlWriter.Create(toFilePath, writerSettings))
        {
            new CustomerAliasXmlEditor(xmlReader, xmlWriter).Process();
        }
    }

    public CustomerAliasXmlEditor(XmlReader reader, XmlWriter writer)
        : base(reader, writer, ShouldTransform, Transform)
    {
    }

    static bool ShouldTransform(XmlReader reader)
    {
        return reader.GetElementName() == customerNamespace + "customer";
    }

    static void Transform(XmlReader from, XmlWriter to)
    {
        var customer = XElement.Load(from);
        var alias = customer.Element(customerNamespace + "alias");
        if (alias != null)
        {
            var name = customer.Element(customerNamespace + "name");
            if (name == null)
            {
                name = new XElement(customerNamespace + "name");
                customer.Add(name);
            }
            alias.Remove();
            name.Add(alias);
        }
        customer.WriteTo(to);
    }
}

Then if fromFileName is the name of your current 3GB XML file and toFileName is the name of the file to which to output the transformed XML, you can do:

var readerSettings = new XmlReaderSettings { IgnoreWhitespace = true };
var writerSettings = new XmlWriterSettings { Indent = false}; // Or true if you prefer.

CustomerAliasXmlEditor.TransformFromTo(fromFileName, readerSettings, toFileName, writerSettings);

Sample working .Net fiddle showing that the XML

<Root>
<Item>
<SubItem>
<customer>
<name><first>Robert</first></name>
<alias>Rob</alias>
</customer>
</SubItem>
</Item>
<Item>
</Root>

Is transformed to

<Root>
  <Item>
    <SubItem>
      <customer>
        <name>
          <first>Robert</first>
          <alias>Rob</alias>
        </name>
      </customer>
    </SubItem>
  </Item>
  <Item>
</Root>