Long time response for query in ef core

Question

I'm creating a select for transactions when group us per user and date, and i need show totalizers also. The result is correct but the time for processing this query is 60s.

I'm using ef core 2.2 and i try using a similar linq expression but the time to response is the same.

public async Task<ListDataPagination<AdvanceIn, TotalizerAdvanceInDate>> AdvanceInDateAsync(int page, int? limit=15)
        {
            IQueryable<AdvanceIn> query;

            var data = new ListDataPagination<AdvanceIn, TotalizerAdvanceInDate>();

            query = Context.SaleTransaction
                .Include(s => s.AdminSale)
                .ThenInclude(a => a.PlaceCategory)
                .Include(s => s.Transaction)
                .Where(s => s.PayIn.Date <= DateTimeOffset.UtcNow.Date.AddDays(6) 
                    && s.Paid == false
                    && s.ProcessingPaid == false
                    && (s.Transaction.Status == TransactionStatus.Authorized || s.Transaction.Status == TransactionStatus.Paid) )
                .GroupBy(s => new { s.AdminSaleId, s.PayIn.Date })
                .Select(s => new AdvanceIn()
                {
                    Count = s.Count(),
                    NetAmountSum = s.Select(t => t.NetAmount).Sum(),
                    GrossAmountSum = s.Select(t => t.Transaction.GrossAmountWithoutSaleDiscount).Sum(),
                    AdminSale = new SaleTransferReadModel {
                        Id = s.Select(t => t.AdminSale).First().Id,
                        PlaceName = s.Select(t => t.AdminSale).First().PlaceName,
                        PlacePhoto = s.Select(t => t.AdminSale).First().PlacePhoto
                    },
                    PayIn = s.Select(t => t.PayIn).First(),
                    SaleTransactionId = String.Join(",", s.Select(x => x.Id.ToString()))
                })
                .OrderBy(s => s.PayIn);

            var list = await query.AsNoTracking().ToListAsync();
            data.Totalizer = new  TotalizerAdvanceInDate {
                Count = list.Sum(t => t.Count),
                NetAmountSum = list.Sum(a => a.NetAmountSum),
                GrossAmountSum = list.Sum(a => a.GrossAmountSum)
            };

            data.Page = page;
            data.TotalItems = list.Count();
            data.TotalPages = data.TotalItems / limit.Value;

            data.Data = query.Skip(limit.Value * page)
                   .Take(limit.Value)
                   .ToList();


            data.PageTotalizer = new TotalizerAdvanceInDate {
                Count = data.Data.Sum(a => a.Count),
                NetAmountSum = data.Data.Sum(a => a.NetAmountSum),
                GrossAmountSum = data.Data.Sum(a => a.GrossAmountSum)
            };

            return data;
        }

I expect reduce time for get this query response.

Answer 1

A few suggestions:

1. As NetMage points out, your first .ToList() call is a major issue. This is materializing the entire set to memory which means pulling all of that data from the DbServer to the web server. As far as I can see that was done just to get a count.

Updated code:

query = Context.SaleTransaction
               .Where(s => s.PayIn.Date <= DateTimeOffset.UtcNow.Date.AddDays(6) 
                    && s.Paid == false
                    && s.ProcessingPaid == false
                    && (s.Transaction.Status == TransactionStatus.Authorized || s.Transaction.Status == TransactionStatus.Paid) )
                .GroupBy(s => new { s.AdminSaleId, s.PayIn.Date });
var rowCount = query.Count();

This will execute one query to get the count of groups without pulling back all of the data.

2. When using projection ( Select ) You do not need to use the Include statements. Those only apply when retrieving the actual entity graph, such as during an Update.

3. Next, once you calculate your paging figures from the Count:

-

data.Data = await query.Select(s => new AdvanceIn()
    {
       Count = s.Count(),
       NetAmountSum = s.Select(t => t.NetAmount).Sum(),
       GrossAmountSum = s.Select(t => t.Transaction.GrossAmountWithoutSaleDiscount).Sum(),
       AdminSale =  s.AdminSale.OrderBy(/* ?? */).Select(a => new SaleTransferReadModel 
       {
           Id = a.Id,
           PlaceName = a.PlaceName,
           PlacePhoto = a.PlacePhoto
       }).First(),
       PayIn = s.Select(t => t.PayIn).First(), // Nope! don't return an Entity. Get a value or create a ViewModel like SaleTransferReadModel.
       SaleTransactionId = String.Join(",", s.Select(x => x.Id.ToString())) // This might not work for EF2SQL... Consider returning the list of IDs and projecting to a Display String in the ViewModel.
    }).OrderBy(s => s.PayIn)
    .Skip(limit.Value * page)
    .Take(limit.Value)
    .ToListAsync();

This will run the second query to pull back just the selected page of data from the database. I'd be considering using async here to allow your web server to process requests while this query is running as it might take a second or so depending on the data involved.

Key changes/notes here: Don't repeat the First call to populate the inner view model. Project it with Select based on the First desired child record. When using First etc. you should always include an OrderBy because you cannot rely on any default ordering.

Next, the code is embedding an Entity, PayIn. Either take a field from PayIn to return or project it into a ViewModel. You don't want to embed any entities in your view model because when the serializer goes to turn it into JSON or whatever, it will touch any properties on PayIn and potentially trip lazy-loads.

Lastly would be the string.Join() on the ID list. This might work, but it may not and EF core has a nasty habit of throwing up a warning that it's going to prematurely materialize a query to 2-step the projection which defeats the entire purpose. I would instead Select a List<string> of IDs into the view model, then expose an extra property on the view model that does a string.Join() .

public ICollection<Guid> SalesTransactionIds { get; set; } // Populate this in your Linq expression...

public string DisplaySalesTransactionIds 
{
   get { return string.Join(",", SalesTransactionIds); }
}

** note: If that gives you any grief about types, you can use SalesTransactionIds.Select(x => x.ToString()) inside of the Join .

When the serializer crawls that, the view/grid should get a string property in the view's model called DisplaySalesTransactionIds which will give you the comma-delimited list.

Have a go with some of those and see if it speeds things up a bit.