Complication in adding and removing object to a large collection in c#

Question

Couple of days ago, I faced a question that I have a collection which is having historical data or very large amount of objects. This collection is exposed to so many threads or clients, So, people might be iterating over it and some might be adding to it and some might be removing to this collection while iteration. So, modification might throw "collection changed exception" in c#.

Now, I need to design a data structure or a collection in c# which fulfills following challenges :

1. You can't copy the collection to different object as Collection is very large, So copying it would cause us lot of memory wastage.

2. while any user adds to collection while iterating the collection, new object should be added in the collection and should not throw any exception and should also be read in the end of the iteration as well as.

3. But in case user removes any item from the collection, then it should throw the exception.

4. Adding, removing and iterating should be thread safe. No race condition should be there.

Answer 1

If you really insist to use a collection and not a database. None of the regular .NET lists will be a good option. In that case you could create your own list type that is optimised for your situation.

It depends on other details (can you work with paging, do you need to acces items by index etc) what technics you could use.

An idea could be that you create one mutable list and only when it change, you create one immutable copy. All the clients use the last immutable copy.