store tabular data other than in data table using c#

Question

i have some tabular data in c# having around 100 K(100000) records which i have to store in memory. What is the best way to store this data other than in table format keeping in mind i have filter out result which is based on some conditions ( like dt.select("field1=1 and...) ,sorting resultset just like SQL table.

kindly suggest any other way to retrieve data.Dictionary is another way but based on field conditions how to retrieve data if using Dictionary or any other collections.

Answer 1

Assuming you go for speed an low on memory consumption, try this:

1. Create a model class which contains a property for each column in the source table. This is your entity.
2. Read from the source table (if it comes from a database, use a DataReader ). Read the data record by record and for each record, create an entity. During reading each field of type string, you can optimize a little bit:
   • Optimize for speed: read the string and put it directly into the property of the entity.
   • Optimize for memory: read the string, use String.Intern on it and put in into the property.
3. Store all these entities in a collection. Here you have two choises:
   • Use a List<Entity> to store it all. You can use LINQ on the list and entities to query your collection. This is rather slow on performance but the best solution for memory.
   • If you know in advance which queries/criteria you are going to use, use one dictionay for a set of criteria. For example. If you have the properties "FirstName" and "LastName", make a dictionary which will store your entity as a value and a Tuple<string, string> which the values of the FirstName and LastName. Now it is extremely fast to query on these values. For sorting, use the SortedDictionary . If a key has duplicates, create a dictionary like this: Dictionary<Tuple<string, string>, List<Entity>> which will store all records with the same matching first- and last names. I know this solution requires more coding, but is pretty fast.

Of course you can keep the DataTable solution. If memory is your only concern, try to make a DataReader -wrapper which will Intern all strings. Wrap your wrapper arround the original DataReader and use it to create / fill the DataTable .