如何优化 foreach 语句的 EF 性能
[英]How to optimize EF performance on foreach statement
问题描述
这是我的代码； 如您所见，我正在为每一行保存更改，但我希望提高性能，因为我每次都有大量数据，例如 50.000 或 100.000 行或更多。 这需要很长时间才能完成。
如何提高我的 EF SaveChanges性能？ 我尝试使用一些第三方库进行批量保存、批量更新，但它没有在数据库中更新。 更新 50.000 行需要 2 小时。 我想改进这种方法的时间。
private void TransferOrders()
{
using (var context = new BbsfDbContext())
{
context.DisableFilter(AbpDataFilters.MayHaveTenant);
context.DisableFilter("LanguageSpecificFilter");
var sapOrders = context.SapOrders
.Where(p => p.VBTYP != null &&
p.VBTYP.ToLower() == OrderDocumentType &&
p.IsRead == false)
//.Where(p => p.VBTYP != null && p.VBTYP.ToLower() == OrderDocumentType && p.Id == 3025)
.Where(p => !ActiveUsersOnly || context.Users.Where(u => u.IsActive).Select(a => a.MainVendor.SapCode).Contains(p.KUNNR))
.OrderBy(p => p.CreatedDate)
.ToList();
if (sapOrders.Any())
{
foreach (var item in sapOrders)
{
try
{
var order = context.Orders.FirstOrDefault(p => p.SapCode == item.VBELN);
var isExist = context.SapOrderDetails.Any(p => p.DOCNUM == item.DOCNUM);
if (isExist)
{
var salesOrganization = context.SalesOrganizations.FirstOrDefault(p => p.SapCode == item.VKORG);
if (salesOrganization == null)
continue;
var distributionChannel = context.DistributionChannels.FirstOrDefault(p => p.SapCode == item.VTWEG);
if (distributionChannel == null)
continue;
var salesDepartment = context.SalesDepartments.FirstOrDefault(p => p.SapCode == item.SPART);
if (salesDepartment == null)
continue;
var salesOffice = context.SalesOffices
.FirstOrDefault(p => p.SapCode == item.VKBUR &&
p.SalesOrganization.Id == salesOrganization.Id &&
p.DistributionChannel.Id == distributionChannel.Id &&
p.SalesDepartment.Id == salesDepartment.Id);
if (salesOffice == null)
continue;
var ordererCustomer = context.Customers
.FirstOrDefault(p => p.SapCode == item.KUNNR &&
p.SalesOrganization.Id == salesOrganization.Id &&
p.DistributionChannel.Id == distributionChannel.Id &&
p.SalesDepartment.Id == salesDepartment.Id &&
p.SalesOffice.Id == salesOffice.Id);
var recipientCustomer = context.Customers
.FirstOrDefault(p => p.SapCode == item.KUNWE &&
p.SalesOrganization.Id == salesOrganization.Id &&
p.DistributionChannel.Id == distributionChannel.Id &&
p.SalesDepartment.Id == salesDepartment.Id &&
p.SalesOffice.Id == salesOffice.Id);
if (recipientCustomer == null)
recipientCustomer = context.Customers
.FirstOrDefault(p => p.SapCode == item.KUNWE &&
p.SalesOrganization.Id == salesOrganization.Id &&
p.DistributionChannel.Id == distributionChannel.Id &&
p.SalesDepartment.Id == salesDepartment.Id &&
p.SalesOffice == null);
if (ordererCustomer == null || recipientCustomer == null)
continue;
if (order == null)
{
order = new Order
{
SapCode = item.VBELN,
SapOrderDate = item.AUDAT,
DocumentType = context.DocumentTypes.FirstOrDefault(p => p.SapCode == item.VBTYP),
SalesDocument = context.SalesDocuments.FirstOrDefault(p => p.SapCode == item.AUART),
BaseAmount = item.NETWR,
TotalTax = item.MWSBT,
Currency = context.CurrencyDefinitions.FirstOrDefault(p => p.SapCode == item.WAERK),
SalesOrganization = salesOrganization,
DistributionChannel = distributionChannel,
SalesDepartment = salesDepartment,
SalesGroup = context.SalesGroups.FirstOrDefault(p => p.SapCode == item.VKGRP && p.SalesOffice.Id == salesOffice.Id),
SalesOffice = salesOffice,
RequestedDeliveryDate = item.VDATU,
SASNo = item.BSTNK,
SASOrderDate = item.BSTDK ?? item.AUDAT,
OrdererCustomer = ordererCustomer,
RecipientCustomer = recipientCustomer,
//PRSDT
Status = OrderStatus.Approved,
Type = OrderType.MainVendor,
DeliveryAddress = context.CustomerAddressBooks.FirstOrDefault(p => p.MainVendor.Id == ordererCustomer.Id && p.SubVendor.Id == recipientCustomer.Id),
CreationTime = DateTime.Now,
LastModificationTime = DateTime.Now,
CreatorUserId = context.Users.First(p => p.UserName == AbpUserBase.AdminUserName).Id,
LastModifierUserId = context.Users.First(p => p.UserName == AbpUserBase.AdminUserName).Id,
IsSubVendorOrder = false,
IsSameDayDelivery = false,
RepresentativeId = context.Users.First(p => p.UserName == AbpUserBase.AdminUserName).Id
//ProductionSite
//RejectionReason =//todo:bu silinmeli iptal kalem bazında burada statu olmalı
};
var savedOrder = context.Orders.Add(order);
context.SaveChanges();
order.SASNo = BbsfConsts.KeasOrderNumberPrefix + savedOrder.Id;
}
else
{
order.SapOrderDate = item.AUDAT;
order.DocumentType = context.DocumentTypes.FirstOrDefault(p => p.SapCode == item.VBTYP);
order.SalesDocument = context.SalesDocuments.FirstOrDefault(p => p.SapCode == item.AUART);
order.BaseAmount = item.NETWR;
order.TotalTax = item.MWSBT;
order.Currency = context.CurrencyDefinitions.FirstOrDefault(p => p.SapCode == item.WAERK);
order.SalesOrganization = salesOrganization;
order.DistributionChannel = distributionChannel;
order.SalesDepartment = salesDepartment;
order.SalesGroup = context.SalesGroups.FirstOrDefault(p => p.SapCode == item.VKGRP && p.SalesOffice.Id == salesOffice.Id);
order.SalesOffice = salesOffice;
order.RequestedDeliveryDate = item.VDATU;
order.SASNo = BbsfConsts.KeasOrderNumberPrefix + order.Id;
//order.SASOrderDate = item.BSTDK.HasValue ? item.BSTDK : item.AUDAT;
order.OrdererCustomer = ordererCustomer;
order.RecipientCustomer = recipientCustomer;
//PRSDT
//order.Status = OrderStatus.Approved;
order.DeliveryAddress = context.CustomerAddressBooks.FirstOrDefault(p => p.MainVendor.Id == ordererCustomer.Id && p.SubVendor.Id == recipientCustomer.Id);
order.LastModifierUserId = context.Users.First(p => p.UserName == AbpUserBase.AdminUserName).Id;
order.LastModificationTime = DateTime.Now;
//ProductionSite
//RejectionReason =//todo:bu silinmeli iptal kalem bazında burada statu olmalı
}
}
else
{
if (order != null)
{
var orderDetails = context.OrderDetails.Where(p => p.OrderId == order.Id).ToList();
orderDetails?.ForEach(p => context.OrderDetails.Remove(p));
context.SaveChanges();
context.Orders.Remove(order);
context.SaveChanges();
}
}
item.IsRead = true;
item.ModifiedDate = DateTime.Now;
context.SaveChanges();
}
catch (Exception ex)
{
logger.Error(ex, MethodBase.GetCurrentMethod().Name + " Error During IDOCOperations " + ex.Message);
continue;
}
}
}
}
}
3 个解决方案
解决方案1
1 2019-11-07 08:33:42
乍一看，似乎可以对不同的操作进行分组，然后在一个步骤中执行。
例如：
- 创建必须使用
item.IsRead = true; item.ModifiedDate = DateTime.Now;item.IsRead = true; item.ModifiedDate = DateTime.Now;, 并在最后一步执行所有操作 - 以同样的方式，创建一个包含所有要删除的订单的列表，然后在最后一步执行
我不知道这是否适用于上下文和/或应用程序，这只是一个想法
解决方案2
0 2019-11-07 08:38:04
解决方案3
0 2019-11-07 09:07:50
如果您可以避免在循环中调用.SaveChanges() ，而是在最后执行它，那么您的情况会好得多，因为您可以避免多次往返数据库。 话虽这么说，如果你循环超过 50,000 个项目，你可能希望在某种程度上对它进行批处理，也许每 1,000 个你调用它。
不幸的是，您需要在保存时将订单创建的结果存储在另一列上，这很复杂。 也许如果您跟踪您正在创建的订单，当您进行批量保存时，您可以立即为刚刚创建的每个订单和另一个.BulkSaveChanges()执行一批SASNo ？
对于您以后的保存更改（删除订单详细信息，保存，删除订单，保存），我看不出需要在多个步骤中执行此操作，但也许我的 EF 生锈了，它会抱怨。 理想情况下，我会删除所有对.SaveChanges()的调用，并在每 1000 次批量操作中执行此操作。
假设BulkSaveChanges可以处理所有这些，以上将显着减少 DB 网络调用的数量。 基本上我的目标是在下面，但归根结底，如果没有 EF，这可能会做得更好/更快。
using (var context = new BbsfDbContext())
{
var sapOrders = ...;
var ordersCreated = new List<..>(); // might wanna initialized this with a size if you have a rough gauge on what % will need creation of loop
//if (sapOrders.Any()) // not needed
//{
foreach (var item in sapOrders.Select((x, index) => new { x, index }))
{
try
{
var order = ...;
var isExist = ...;
if (isExist)
{
// ...
if (order == null)
{
order = new Order { ... };
var savedOrder = context.Orders.Add(order);
//context.SaveChanges();
//order.SASNo = BbsfConsts.KeasOrderNumberPrefix + savedOrder.Id;
ordersCreated.Add(order);
}
else
{
// Do updates
// ...
}
}
else
{
//if (order != null) // shouldn't need this
//{
var orderDetails = context.OrderDetails.Where(p => p.OrderId == order.Id).ToList();
orderDetails?.ForEach(p => context.OrderDetails.Remove(p));
//context.SaveChanges();
context.Orders.Remove(order);
//context.SaveChanges();
//}
}
// ...
if (index % 1000 == 0)
{
context.BulkSaveChanges(); // bulk save of 1000 loops of changes
foreach (var orderCreated in ordersCreated)
{
orderCreated.SASNo = BbsfConsts.KeasOrderNumberPrefix + savedOrder.Id;
}
context.BulkSaveChanges(); // bulk save of x num of SASNo sets
}
}
catch (Exception ex)
{
// ...
}
}
}
}
}
在 foreach 中使用 EF Core 包含方法可能是一个性能问题
[英]Is using EF Core include method inside a foreach can be a performance issue
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.