Best way to manage a large amount of data in memory?

Question

I'm trying to find the best way to manage a large amount of data in memory using C# without access a database. (the db will be used to store just part of these information when they become final)

When I say "large amount of data" I'm talking about hundreds of megabytes and I will like to manage a complex structure, not only something like a table with millions of records.

I need to search inside them as fast as possible and I need to be able to remove part of them when they become obsolete.

Luckily I can split this data in groups that don't need to be related in some way...so I don't need to find or update a row between millions, but something like find a group between lets say 50,000 other, search, add and remove data in this group and delete all the group when it become obsolete.

I have some projects that already manage data in memory but nothing so huge so I don't know if these methods are applicable also in this situazion: -I used the .Net cache object but I never worked with more that 10 or 20 megabytes -private static List data = new List(); on which I stored groups of data in xml format, but I never worked with more than a couple of megabyte -datatable objects, one for group and also in this last case I never worked with more than 10 megabytes and I had problems to manage the access because datatables aren't thread safe

What could be the best way to manage this kind of situation? There is any kind of limit of Windows or of the .Net framework that can create me problems?

Answer 1

You should be fine with using your memory to store data in a data structure and playing around with data. The trade off is that the memory will not be available to other applications running on same server. Also when you try to commit the data from Memory to disk/DB the time taken for large data is more.

Depending on your data, the structure has to be defined. If the data is not inter related you should create different objects for each entity.

You will also need to device a strategy to upate/refresh your cache. Either it can be hourly, daily, weekly; depending on your needs.