Queries on IQueryable with property name and type at runtime

Question

I need to do queries with knowing the property name and type at runtime. I've used reflection on IEnumerable<> but will there be issues with performance because of this?

I would like to know if there is a better way of doing this with IQueryable<> ? I've looked a bit into Expressions but I'm not quite sure how to do it.

Edit:

At the moment it doesn't seem to be a performance issue, but I haven't tested yet with very big workloads.

I need to search in multiple fields of different types known at runtime.

var cosmosClient = new DocumentClient(new Uri(cosmosDBEndpointUrl), cosmosDBAuthorizationKey);
var feedOptions = new FeedOptions { MaxItemCount = -1, EnableCrossPartitionQuery = true };
var objects = cosmosClient.CreateDocumentQuery<MyObject>(collectionLink, feedOptions).AsEnumerable();

if (!string.IsNullOrEmpty(searchQuery))
{         
    var predicate = PredicateBuilder.New<MyObject>(); 

    foreach (var fieldToSearch in fieldsToSearch)
    {
      predicate = predicate.Or(x => x.GetPropertyValue(fieldToSearch).CheckDateTime().ToString()
                                     .Contains(searchQuery, StringComparison.InvariantCultureIgnoreCase));
      objects = objects.Where(predicate);
    }
}

objects = objects.Skip(index)
                 .Take(pageSize);

return objects.ToList();

And this helper method:

public static object GetPropertyValue(this object obj, string propertyName)
{
    foreach (var part in propertyName.Split('.'))
    {
        if (obj == null) { return null; }

        Type type = obj.GetType();
        PropertyInfo info = type.GetProperty(part, BindingFlags.IgnoreCase | BindingFlags.Public | BindingFlags.Instance);
        if (info == null) { return null; }

        obj = info.GetValue(obj, null);
    }
    return obj;
}

I don't know in this particular case if there is a better way of doing this.

Answer 1

So in this case I do think you want IQueryable instead of IEnumerable . The easiest way to remember is that IEnumerable loads all the data into memory and then runs the rest of your LINQ query while IQueryable tries to run your filters on the source if it can then loads the resulting data into memory. You do need to be using an ORM that supports IQueriable like LINQ to SQL or Entity Framework. In the case of CosmosDB, the SDK is built on top of LINQ to SQL.

By using IEnumerable you are loading all the data from Cosmos into memory and then applying your filters and pagination. It's fine while you only have a few records, but as your dataset grows you can probably imagine how this can become a major problem. Generally you want to do as much work as possible in the database and only return a minimal number of results. That is going to far outweigh any performance considerations from using reflection to build your predicates.

One of the best parts of IQueryable is that it inherits from IEnumerable so anything that works with IEnumerable will work with IQueriable . All you should need to do is get rid of your AsEnumerable() .

---

It is important to note that not all LINQ operators are automatically supported, and as soon as the system hits one it can't translate it executes what it has so far and does the rest in memory. There is a list of available operators for CosmosDB in the documentation.

For your query, the big one is going to be that, while your Where() clause is supported, Skip() and Take() currently aren't. That means that each time this method is executed all the results will be returned from CosmosDB and then the pagination will be evaluated.

There are a couple ways to deal with pagination in the SDK. The currently supported way is to set the MaxItemCount inside your FeedOptions to your page size. Instead of the Skip() function, the current system uses continuation tokens. In order to access the token you can use AsDocumentQuery() . Since you have to step through each page to the next one, caching the tokens can be very helpful- jumping around is tough.

The second option is to use v3 of the .Net SDK. It is currently in preview, but available on Nuget . A few months ago skip/take was enabled . In that case, everything up until you call ToList() should be translated to SQL and evaluated within CosmosDB.