Create indexed class collection

Question

I'm looking to create a collection of c# classes which I need to index in order to speed up access to certain properties. The catch is, the class contains a lot of references to existing services/classes which aren't easily serialized/deserialized, so a database wont suffice.

class ComplexTestClass
{
    public MyService Service;
    public Guid Id;
    public ComplexObject Object;
}

using linq, I can query a collection like so:

complexTestClassCollection.where(complexClass => complexClass.Object.Type == "TestType");

When my collection grows (It does get rather large), these linq queries are horribly inefficient. I'm looking to create some type of index, so that I can do lookups on properties to avoid linq queries all together.

My search lead me to LiteDB which had an impressive way to index/query a collection, but required the collection to be serialized/deserialized which is where I'm running into issues currently (MyService is not able to be serialized without massive rewrites).

To clarify, I need more complex indexing like what LiteDB does. An example query might be:

complexTestClassCollection.where(complexClass => complexClass.Object.Type == "TestType" && complexClass.Object.ParentObject == "TestParent" && complexClass.Object.Members.Contains(4));

In the above case, I'm ok with complexClass.Object.Members.Contains(4) not being indexed, as complexClass.Object.Type == "TestType" && complexClass.Object.ParentObject == "TestParent" dims the result set down to < 10 elements, and running linq on that collection is acceptable.

Answer 1

Linq queries follow deferred execution, so unless you do a .ToList( ) or ToCollection( ), it will not bring in all the results. Also it depends on kind of search you are performing. I would prefer First( ) extention in linq to do an early exit.

You could also put data in database and use IQueryable for good performance.

In long term, it would be maintainable and scalable, to create a query engine that would translate your queries into sql statements, you can then scale horizontally by refactoring it to different classes and querying against them, instead of creating a big collection and hardcoding against types and properties.