从API提取XML并将其存储在本地的最有效方法是什么？

Question

I am trying to find the fastest way to read XML from the merriam webster dictionary, and store it to a local file for later use. Below, I try to implement a module which does a few things:

1. Read 2000 words from a local directory
2. Look up each of the words in the merriam dictionary using the API
3. Store the definition(s) in a local XML for later use.

Im not sure if making an XML is the best way to store this data, but it seemed like the simplest thing to do. At first, I thought I would do it in different steps. (1. Look up word, store word and definitions into data structure. 2. Dump all data into XML.) However, this poses a problem, because it just too much stuff to store on the runtime(call) stack.

So, in this scenario, I try to speed things up by looking up each word and then saving it to the xml one by one. This, however, is also a slow method. Its taking me up around 10 minutes per 500-600 words.

public void load_module() // stores words/definitions into xml file
    { // 1. Pick up word from text file     2. Look up word's definition    3. Store in Xml 
        string workdirect = Directory.GetCurrentDirectory();
        workdirect = workdirect.Substring(0, workdirect.LastIndexOf("bin"));
        workdirect += "words1.txt";
        using (StreamReader read = new StreamReader(workdirect)) // 1. Pick up word from text file 
        {
            while (!read.EndOfStream)
            {
                string line = read.ReadLine(); 
                var definitions = load(line.ToLower());    // 2. Retrieve Words Definitions

                store_xml(line, definitions);
                wordlist.Add(line);
            }
        }
    }

    public List<string> load(string word)
    {
        XmlDocument doc = new XmlDocument();

        List<string> definitions = new List<string>();
        XmlNodeList node = null;

        doc.Load("http://www.dictionaryapi.com/api/v1/references/collegiate/xml/"+word+"?key=*****************"); // Asteriks to hide the actual API key

        if (doc.SelectSingleNode("entry_list").SelectSingleNode("entry").SelectSingleNode("def") == null)
        {
            return definitions;
        }
        node = doc.SelectSingleNode("entry_list").SelectSingleNode("entry").SelectSingleNode("def").SelectNodes("dt");

        // TO DO : implement definitions if there is no node "def" in first node entry "entry_list"

        foreach (XmlNode item in node)
        {
            definitions.Add(item.InnerXml.ToString().ToLower());
        }


        return definitions;

    }

    public void store_xml(string word, List<string> definitions)
    {
        string local = Directory.GetCurrentDirectory();
        string name = "dictionary_word.xml";
        local = local.Substring(0, local.LastIndexOf("bin"));
        bool exists = File.Exists(local + name);

        if (exists)
        {
            XmlDocument doc = new XmlDocument();
            doc.Load(local + name);
            XmlElement wordindoc = doc.CreateElement("Word");
            wordindoc.SetAttribute("xmlns", word);
            XmlElement defs = doc.CreateElement("Definitions");
            foreach (var item in definitions)
            {
                XmlElement def = doc.CreateElement("Definition");
                def.InnerText = item;
                defs.AppendChild(def);
            }
            wordindoc.AppendChild(defs);
            doc.DocumentElement.AppendChild(wordindoc);
            doc.Save(local+name);
        }
        else
        {
            using (XmlWriter writer = XmlWriter.Create(@local + name))
            {
                writer.WriteStartDocument();

                writer.WriteStartElement("Dictionary");

                writer.WriteStartElement("Word", word);

                writer.WriteStartElement("Definitions");
                foreach (var def in definitions)
                {
                    writer.WriteElementString("Definition", def);
                }
                writer.WriteEndElement();
                writer.WriteEndElement();

                writer.WriteEndElement();
                writer.WriteEndDocument();
            }
        }           
    }
}

Answer 1

When handling large amounts of data that need to be exported to XML, I would normally keep the data in memory as a collection of custom objects rather than as an XMLDocument:

public class Definition
{
    public string Word { get; set; }
    public string Definition { get; set; }
}

I would then use XMLWriter to write the collection to the XML file:

XmlWriterSettings settings = new XmlWriterSettings();
settings.Indent = true;
settings.IndentChars = ("    ");
settings.Encoding = Encoding.UTF8;
using (XmlWriter writer = XmlWriter.Create("C:\output\output.xml", settings))
{
    writer.WriteStartDocument();
    // TODO - use XMLWriter functions to write out each word and definition
    writer.Flush();
}

If you are still short on memory, you might be able to write out the XML in batches (eg every 500 definitions).

I found the Microsoft article on Improving XML Performance a very useful reference, particularly the section on Design Considerations.