What should I use to temporary store XML data that will be reused?

Question

I have a 47mb XML file that I have to read over and over and over on my application but mostly it only wants 1 node defined by a given term every now and then.

Example:

<customers>
    <customer>
        <id>someid</id>
        <name>somename</name>
    </customer>
    <customer>
        <id>someid</id>
        <name>somename</name>
    </customer>

    ...

    <customer>
        <id>someid</id>
        <name>somename</name>
    </customer>
    <customer>
        <id>someid</id>
        <name>somename</name>
    </customer>
</customers>

So I was thinking on loading the xml to a Dictionary or List once the programs load up and keep it there to reuse until the application is closed.

• What I wanted to know what would be the best type of stored for it ?

If I have to use a dictionary for example:

private Dictionary<int,string> myCustomerList = new Dictionary<int,string>();

which I could use later to get the information I need like:

if (myCustomerList.ContainsKey(keyX))
    myValue = myCustomerList[keyX];

• Is this a bad way to go or what should I be looking at ?

Answer 1

There is nothing functionally wrong with your approach, but what happens when more information is added to the Customer? For example, you might have a "location" element:

   <customer>
        <id>someid</id>
        <name>somename</name>
        <location>somename</location>
    </customer>

Also consider the scenario where you move your data to a database - you will have to rewrite a lot of code.

It is generally considered bad practice to work directly with XML objects through the duration of your application. The "correct" approach is to first load your XML data into Object Oriented objects.

You should create a class named Customer , and use List<Customer> customers =new List<Customer>() to store the objects. You can then use customers.Find(x=>x.Id == myId) to locate your objects.

Remember that if you store your objects in memory, your application will use at least 47mb of RAM. This could potentially grow as your business attains more customers. Perhaps you should look at lazy-loading your customers (retrieving them from the XML file when they are required) - this will be slower so it really depends on your throughput and volume requirements.

Answer 2

Your approch is quite right. if in future when more information is added to customer class then use list generic class otherwise your approch is right.

List - one of my favs - can be used with generics, so you can have a strongly typed array, eg List. Other than that, acts very much like ArrayList.

Dictionary - same as above only strongly typed via generics, such as Dictionary

use List and Dictionary all the time.

Answer 3

i would say 47 mb of a file, loading all at once in a datatable would be very easy to do and also good for searching.

You should consider WeakReferences to store this datatable in memory.

Answer 4

47MB is fairly large data for an app to keep in memory all the time.

However, you can combine XmlReader with XNode.ReadFrom to read large Xmls with less memory.

You can use this method to write a method that returns a collection of nodes, yielding each node as the node is read from the reader. This method enables you to process arbitrarily large XML files with a very small memory footprint.