在实体框架中处理大数据的最佳方式

Question

Hi I am using EF6 where I have large data to select from database where my query as follows:

var EntityInfo = _contextRegister.Entities.Where(x => x.IsDeleted != true).ToList();
foreach (var itm in EntityInfo)
{
    Entity.Entity entity = new Entity.Entity();
    entity.MainActivityId = itm.MainActivityId;
    entity.SubGroupId = itm.SubGroupId;
    entity.Id = Convert.ToInt32(itm.Id);
-------
    entity.UAECityRegion = _contextFRAMEWORK.UAECityRegions.Where(m => m.Id == itm.UAECityRegionId).Select(m => m.RegionName).FirstOrDefault();
    var voucherstautus = _contextRegister.EPayVoucherDatas.Where(m => m.EntityId == itm.Id).ToList();
    foreach (var item in voucherstautus)
    {
        if (item.VoucherStatus == 10)
        {
            entity.PaymentStatus = Convert.ToInt32(item.VoucherStatus);
            break;
        }
     }

     entityList.Add(entity);
}

This query taking more than 10mins where how can I make this query better or should I use stored procedure instead of framework?

Answer 1

As far as I can see there is a 1:1 relationship between Entities and EPayVoucherDatas (otherwise your code _contextRegister.EPayVoucherDatas.Where(m => m.EntityId == itm.Id) wouldn't work). So at first ensure you have modeled this relationship in your EF model .

If that is true you could write something like this:

_contextRegister.EPayVoucherDatas.Where(m => !m.Entity.IsDeleted);

That already eliminates a part of your problems.

Another part is your usage of another database _contextFRAMEWORK , which seems to contains some other data under the same Ids. In your current implementation you ask for a specific region for each item separately. But depending on the size of this table it could be wiser to pull this whole table once (outside of the for-loop) and make the lookup locally. Also you have this if-statment that sets the PaymentStatus to 10 if the VoucherStatus is also 10. What is in all other cases`? Maybe you can always map the PaymentStatus from the VoucherStatus?

A lot of question and not a real answer, cause as usual it depends . Try to minimize the amount of calls to the database and check how many entities each call returns. Filter by some conditions as early as possible to minimize the amount of data that the server returns and try to make as few connections as possible to the server. Even if that results in quite complex queries. This is not a problem for a SQL server normally, cause that's it for what it was made for.

Answer 2

1) Check if the DataBase Indexes are created for each field that is used in the WHERE conditions, for example:

x.IsDeleted
m.Id 
m.EntityId

You can create indexes using the Model Builder:

modelBuilder.Entity<Person>()
    .HasIndex(p => p.Name)
    .IsUnique();

See https://stackoverflow.com/a/47031294/194717

You can also create indexes with IndexAttribute

public class Blog
{
    public int Id { get; set; }
    public string Title { get; set; }

    [Index]
    public int Rating { get; set; }

    public virtual ICollection<Post> Posts { get; set; }
}

See https://blog.oneunicorn.com/2014/02/15/ef-6-1-creating-indexes-with-indexattribute/

2) You shouldn't hit the database two times for each Entities record. Try to write it as a JOIN instead ( .Include() ).