Entity Framework generic expression to selectively hide property

Question

I would like to selectively ignore a property from a table. I have an API which exposes the following methods.

public interface IReadService 
{
   FullDTO Get();
   HeaderDTO[] GetList();
}

My data structure looks like so:

public ServiceDTO : ServiceHeaderDTO 
{
    public string LargeXMLData { get; set; }
}

public ServiceHeaderDTO 
{
    public int Id { get; set; }    
    public string Description { get; set; }
    //.... Other properties
}

I have a few services which have similar issues, So I would like to be able to ignore the XML property in some cases, so I'm not using extra time to send a large string property which will be ignored.

Normally you might write something like this to hide a property

 var entities = context.Services.Select(x => 
    new Service { Id = Id, Description = Description, LargeXMLData = "" }).ToArray();

 var dtos = this.AdaptToDTO(entities);

Now this would be fine if I had to do this in a single service, but when you have 20 services duplicating the logic it gets annoying.

I would like the be able to just say:

 var entities = context.Services.Excluding(x => x.LargeXMLData).ToArray();
var dtos = this.AdaptToHeaderDTO(entities);

Edit: I'm not using automapper. Alot of our code has mappings which cannot translate to expressions. I do not want to have to specify maps

Is there a simple way I can exclude a property from a query? Without having to manually build maps.

Preferably a way which uses the existing mappings internal to EF which maps the entity to the db object

Answer 1

Normally you might write something like this to hide a property

var entities = context.Services.Select(x => new Service { Id = Id, Description = Description, LargeXMLData = "" })

If you can do that manually, it should be doable automatically using the exact same concept, with little reflection and Expression APIs.

But note that this woult work only for EF Core, since EF6 does not support projecting to entity types, like new Service {... } here, and projecting to dynamic types at runtime is not trivial and also will break the DTO mapping.

With that being said, following is a sample implementation of the aforementioned concept:

public static partial class QueryableExtensions
{
    public static IQueryable<T> Excluding<T>(this IQueryable<T> source, params Expression<Func<T, object>>[] excludeProperties)
    {
        var excludeMembers = excludeProperties
            .Select(p => ExtractMember(p.Body).Name)
            .ToList();
        if (excludeMembers.Count == 0) return source;
        // Build selector like (T e) => new T { Prop1 = e.Prop1, Prop2 = e.Prop2, ... }
        // for each public property of T not included in the excludeMembers list,
        // which then will be used as argument for LINQ Select
        var parameter = Expression.Parameter(typeof(T), "e");
        var bindings = typeof(T).GetProperties()
            .Where(p => p.CanWrite && !excludeMembers.Contains(p.Name))
            .Select(p => Expression.Bind(p, Expression.MakeMemberAccess(parameter, p)));
        var body = Expression.MemberInit(Expression.New(typeof(T)), bindings);
        var selector = Expression.Lambda<Func<T, T>>(body, parameter);
        return source.Select(selector);
    }

    static MemberInfo ExtractMember(Expression source)
    {
        // Remove Convert if present (for value type properties cast to object)
        if (source.NodeType == ExpressionType.Convert)
            source = ((UnaryExpression)source).Operand;
        return ((MemberExpression)source).Member;
    }
}

The usage would be exactly as desired:

var entities = context.Services.Excluding(x => x.LargeXMLData).ToArray();

The problem with this though is that it will automatically "include" navigation properties and/or unmapped properties.

So it would be better to use EF model metadata instead of reflection. The problem is that currently EF Core does not provide a good public way of plugging into their infrastructure, or to get access to DbContext (thus Model ) from IQueryble , so it has to be passed as argument to the custom method:

public static IQueryable<T> Excluding<T>(this IQueryable<T> source, DbContext context, params Expression<Func<T, object>>[] excludeProperties)
{
    var excludeMembers = excludeProperties
        .Select(p => ExtractMember(p.Body).Name)
        .ToList();
    if (excludeMembers.Count == 0) return source;
    // Build selector like (T e) => new T { Prop1 = e.Prop1, Prop2 = e.Prop2, ... }
    // for each property of T not included in the excludeMembers list,
    // which then will be used as argument for LINQ Select
    var parameter = Expression.Parameter(typeof(T), "e");
    var bindings = context.Model.FindEntityType(typeof(T)).GetProperties()
        .Where(p => p.PropertyInfo != null && !excludeMembers.Contains(p.Name))
        .Select(p => Expression.Bind(p.PropertyInfo, Expression.MakeMemberAccess(parameter, p.PropertyInfo)));
    var body = Expression.MemberInit(Expression.New(typeof(T)), bindings);
    var selector = Expression.Lambda<Func<T, T>>(body, parameter);
    return source.Select(selector);
}

which makes the usage not so elegant (but doing the job):

var entities = context.Services.Excluding(context, x => x.LargeXMLData).ToArray();

Now the only remaining potential problem are shadow properties, but they cannot be handled with projection, so this technique simply cannot be used for entities with shadow properties.

Finally, the pure EF Core alternative of the above is to put the LargeXMLData into separate single property "entity" and use table splitting to map it to the same table. Then you can use the regular Include method to include it where needed (by default it would be excluded).

Answer 2

I needed to double-check this before answering, but are you using Automapper or some other mapping provider for the ProjectTo implementation? Automapper's ProjectTo extension method requires a mapper configuration, so it may be that your mapping implementation is materializing the entities prematurely.

With Automapper, your example projecting to a DTO that does not contain the large XML field would result in a query to the database that does not return the large XML without needing any new "Exclude" method.

For instance, if I were to use:

var config = new MappingConfiguration<Service, ServiceHeaderDTO>();
var services = context.Services
    .ProjectTo<ServiceHeaderDTO>(config)
    .ToList();

The resulting SQL would not return the XMLData because ServiceHeaderDTO does not request it. It is equivalent to doing:

var services = context.Services
    .Select(x => new ServiceHeaderDTO
    {
        ServiceId = x.ServiceId,
        // ... remaining fields, excluding the XML Data
    }).ToList();

As long as I don't reference x.LargeXMLData , it will not be returned by my resulting query. Where you can run into big data coming back is if something like the following happens behind the scenes:

var services = context.Services
    .ToList()
    .Select(x => new ServiceHeaderDTO
    {
        ServiceId = x.ServiceId,
        // ... remaining fields, excluding the XML Data
    }).ToList();

That extra .ToList() call will materialize the complete Service entity to memory including the XMLData field. Now Automapper's ProjectTo does not work against IEnumerable , only IQueryable so it is unlikely that any query fed to it was doing this, but if you are using a home-grown mapping implementation where ProjectTo is materializing the entities before mapping, then I would strongly recommend using Automapper as it's IQueryable implementation avoids this problem for you automatically.

Edit: Tested with EF Core and Automapper just in case the behaviour changed, but it also excludes anything not referenced in the mapped DTO.