Entity Framework Core Include then Take 2

Question

For learning purpose i made a console application using EFCore 3.1

i have relationship like this:

Person belongs To one Type

Person have many Queries

Query belongs to one Person

Query belongs to a Location

this Console application is connected to a Mysql DB Scaffolded into Entities with Pomelo.

Now i need to get every Persons from the DB then his related Type and Last Queries ordered by date, each person have hundreds queries and there is thousands persons on this db, so if i try to

using (var ctx = new DBContext) 
{
    var persons = ctx.Persons
                  .Include(p => p.Type)
                  .Include(p => p.Queries).ThenInclude(q => q.Location)
}

it will timeout because he's trying to get every queries for each person, like it should be, because i read somewhere in another answer that with the include it's all or nothing.

In my DB i have a double linked view for this purpose, one for get the max_query_id (person_maxquery) for each person_id and one for joining this 4 tables

SELECT p.*, ty.*, pq.*, loc.* FROM persons p
LEFT JOIN person_maxquery max ON p.id = max.person_id
LEFT JOIN person_queries pq ON pq.id = max.query_id
LEFT JOIN locations loc ON loc.id = pq.location_id
LEFT JOIN types ty ON ty.id = p.type_id

I want to achieve the same thing in EF but i didn't know if this is possible, could i filter data got from DB? Should i "make" my object using the view manually assigning every value to related model value?

Answer 1

Queries that pull sets of data rarely need the entire details from the applicable entities. You can mitigate this and filter data by projecting the entities to a view model structure that reflects what data you really need.

using (var ctx = new DBContext) 
{
    var persons = ctx.Persons
        .Select(x => new PersonViewModel
        {
           PersonId = x.PersonId,
           Name = x.Name,
           TypeName = x.Type.Name,
           RecentQueries = x.Queries
               .OrderByDecending(q => q.QueryDate)
               .Select(q => new QueryViewModel
               {
                  QueryId = q.QueryId,
                  Name = q.Name,
                  LocationName = q.Location.Name
                  // ... etc.
               }).Take(2).ToList()
        }).ToList();
    //...
}

This approach helps build more efficient queries with just the data you need. Pulling full entity graphs is more of a per-item request such as when applying an update to a person and their related entities rather than trying to pull back all people with all, or a subset of their related entities.

Edit: Example lifecycle of some data between server and view:

Part 1... List a bunch of people. These are summaries, enough data to satisfy a summary list view.

public IEnumerable<PersonSummaryViewModel> GetPeople(/*criteria*/)
{
    using (var ctx = new DBContext) 
    {
        var people = ctx.Persons
            .Where( /* based on criteria */ )
            .Select(x => new PersonSummaryViewModel
            {
               PersonId = x.PersonId,
               Name = x.Name,
               TypeName = x.Type.Name,
               RecentQueries = x.Queries
                   .OrderByDecending(q => q.QueryDate)
                   .Select(q => new QuerySummaryViewModel
                   {
                      QueryId = q.QueryId,
                      Name = q.Name,
                      LocationName = q.Location.Name
                      // ... etc.
                   }).Take(2).ToList()
            }).ToList();
        //...
        return people;
    }
}

Part 2: Get a Person Detailed Model. Once a user selects a person, we will want to pull back more data. This view model can have a lot more fields and relationships to satisfy the view. But even here we may want to exclude data that is not commonly viewed, (such as in expandable regions or tabs) that can be pulled back on demand with Ajax calls if/when the user wants.

public PersonDetailViewModel> GetPerson(int personId)
{
    using (var ctx = new DBContext) 
    {
        var person = ctx.Persons
            .Select(x => new PersonDetailViewModel
            {
               PersonId = x.PersonId,
               Name = x.Name,
               TypeName = x.Type.Name,
               // ... Can load all visible properties and initial, important related data...
            }).Single(x => x.PersonId == personId);
        //...
        return person;
    }
}

Part 3: Example, Update a Person

public void UpdatePerson(UpdatePersonViewModel updatePersonModel)
{
    if (updatePersonModel == null)
       throw new ArgumentNullException("updatePersionModel");

    // TODO: Validate updateModel, ensure everything is Ok, not tampered/invalid.
    using (var ctx = new DBContext) 
    {
        var person = ctx.Persons
            // can .Include() any related entities that can be updated here...
            .Where( x => x.PersonId == updatePersonModel.PersonId )
            .Single();
        person.Name = updatePersonModel.Name;
        // etc.
        ctx.SaveChanges();
    }
}

Part 4: Example, add a Query.

public void AddQuery(int personId, AddQueryViewModel queryModel)
{
    if (queryModel == null)
       throw new ArgumentNullException("queryModel");

    // TODO: Validate queryModel, ensure everything is Ok, not tampered/invalid.
    using (var ctx = new DBContext) 
    {
        var person = ctx.Persons
            .Include( x => x.Queries )
            .Where( x => x.PersonId == personId )
            .Single();

        // TODO: Maybe check for duplicate query already on Person, etc.
        var query = new Query
        {
            // copy data from view model
        };
        person.Queries.Add(query);
        ctx.SaveChanges();
    }
}

The view models (or DTOs) only relate to transmitting data between the server and the client. When we get data back to the server and want to update entities, we load those entities. By using these view models we reduce the amount of data sent between client and server (just the fields needed, not entire entity graphs) which means faster code and less data over the wire. We don't expose more about our entity structure than the client needs to know, and we don't risk simply overwriting data coming back by something like Attach + EntityState.Modified + SaveChanges where the data could be stale (someone else modified since that copy was taken) or the data coming back could be tampered with. View models are single purpose. We could use a PersonDetailViewModel in an UpdatePerson type operation but our update may only apply to a select few properties so there is the overhead of sending everything back to the server, and no implied restriction on what should, and should not be allowed to be updated. (Especially if you use Automapper or such to help copy fields back and forth between entities and view models)

In cases where we just want to interact with some entity data, without sending it back to the client, we don't need view models/DTOs, we can just selectively retrieve fields from the entire entity graph using Anonymous Types. For instance if I just want to check if a person has a certain matching criteria and summarize the query names that match:

var queryNames = ctx.Persons
    .Where(x => x.PersonId == personId)
    .SelectMany(x => Queries.Select( q => new  
    {
       q.QueryId,
       q.Name
    }).ToList();
var message = string.Join(", ", 
    queryNames.Select(x => string.Format("{0} ({1})", x.Name, x.QueryId)));

Just as a simple example with an anonymous type to return a structure that can be used by further code without the need for a view model / DTO. We're not passing the entity data back, but perhaps just inspecting values to determine a course of action, or composing something like a message string.