[英]Speed up the insert of 70,000 rows (SProc, C#)
I have a list of 70,000 custom objects in C#, that hold four values. 我在C#中有70,000个自定义对象的列表,其中包含四个值。 I need to insert all of these into a SQL Server table, if they don't already exist in a different table (same DB).
我需要将所有这些插入到SQL Server表中(如果它们尚未存在于另一个表(相同的数据库)中)。 I currently have a single SProc that inserts an individual record, with a
foreach
loop calling the SProc for each item in my list. 我目前只有一个SProc插入一条单独的记录,并且有一个
foreach
循环为列表中的每个项目调用SProc。 Inserting 70,000 is taking about 14 minutes, which feels very slow to me, so is there a better approach? 插入70,000大约需要14分钟,这对我来说感觉很慢,是否有更好的方法?
The rough SProc is: 粗略的SProc是:
CREATE Procedure [dbo].[up_UpdateOverrideTable](
@ItemId varchar(100),
@ItemValue1 decimal(8,5),
@ItemValue2 decimal(8,5),
@Source varchar(2)
)
AS
BEGIN
DECLARE @LocItemId varchar(100)
SET @LocItemId = @ItemId
DECLARE @LocItemValue1 decimal(8,5)
SET @LocItemValue1 = @ItemValue1
DECLARE @LocItemValue2 decimal(8,5)
SET @LocItemValue2 = @ItemValue2
DECLARE @LocSource varchar(2)
SET @LocSource = @Source
DELETE FROM OverrideTable WHERE ItemId = @LocItemId
IF EXISTS (SELECT ItemId FROM InitialTable WHERE ItemId = @LocItemId)
INSERT INTO OverrideTable VALUES (@LocItemId, @LocItemValue1, @LocItemValue2, @LocSource)
END
GO
The C# that calls this is also below: 调用此代码的C#也在下面:
using (SqlConnection conn = GetNewConnection()){
conn.Open();
using (var tran = conn.BeginTransaction()){
using (SqlCommand cmd = new SqlCommand("up_UpdateOverrideTable", conn, tran)){
cmd.CommandType = CommandType.StoredProcedure;
try{
foreach (var item in overrides){
cmd.Parameters.Clear();
// add parameters here
var response = cmd.ExecuteNonQuery();
completedInserts.Add(new OverrideItem(item.Id, Convert.ToBoolean(response)));
}
}
catch (Exception ex){
tran.Rollback();
throw;
}
tran.Commit();
conn.Close();
}
}
}
Is there something I'm missing here, or is this the best I can hope for? 我在这里想念的东西还是这是我所希望的最好的东西? I'm wondering whether creating a
DataTable
in C# would do the trick? 我想知道在C#中创建
DataTable
是否可以解决问题?
I'm wondering whether creating a DataTable in C# would do the trick?
我想知道在C#中创建DataTable是否可以解决问题?
Yes, using a DataTable
in C# is a step towards speeding up the operation. 是的,在C#中使用
DataTable
是加快操作速度的一步。 You then need to use it as a data source for SqlBulkCopy
as Matt suggested in comments. 然后,您需要按照注释中的Matt要求将其用作
SqlBulkCopy
的数据源。
DataTable source = null;//your data source as a DataTable
SqlBulkCopy bulkCopy = new SqlBulkCopy("your connection string");
bulkCopy.DestinationTableName = "your target table name";
await bulkCopy.WriteToServerAsync(source);
You have few sql commands which can slow down the procedure response.. 您只有几个sql命令,这些命令可能会减慢过程响应。
DELETE FROM OverrideTable WHERE ItemId = @LocItemId
IF EXISTS (SELECT ItemId FROM InitialTable WHERE ItemId = @LocItemId)
(Make sure you have index on ItemId) IF EXISTS (SELECT ItemId FROM InitialTable WHERE ItemId = @LocItemId)
(确保在ItemId上具有索引) Unless optimization on db level is not done, C# code can't do anything. 除非未在数据库级别进行优化,否则C#代码将无法执行任何操作。
And regarding c# code, I think you should create command inside foreach loop. 关于c#代码,我认为您应该在foreach循环内创建命令。
try{
foreach (var item in overrides)
{
using (SqlCommand cmd = new SqlCommand("up_UpdateOverrideTable", conn, tran))
{
cmd.CommandType = CommandType.StoredProcedure;
// add parameters here
var response = cmd.ExecuteNonQuery();
completedInserts.Add(new OverrideItem(item.Id, Convert.ToBoolean(response)));
}
}
}
catch (Exception ex)
{
tran.Rollback();
throw;
}
tran.Commit();
conn.Close();
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.