[英]SqlDataAdapter.Update() of 2 million records is extremely slow
I have a customer that wants to import his sub-customers pricetools (more that 2.000.000 records) every day into a SQL Server database (and yeah....there are more than 900.000 rows of changes every day).我有一个客户想要每天将他的子客户 pricetools(超过 2.000.000 条记录)导入 SQL Server 数据库(是的......每天有超过 900.000 行的更改)。
The data is provided in CSV format (not in RFC-4180 standard ç_ç, but nvm) and can be an Insert, Delete or Update of data.数据以 CSV 格式提供(不在 RFC-4180 标准 ç_ç 中,而是在 nvm 中)并且可以是数据的插入、删除或更新。
My problem is that the insert of the data inside the database take more than 1 night to end and I need to speed it up.我的问题是在数据库中插入数据需要1个多晚上才能结束,我需要加快速度。
What I'm doing at the moment is:我目前正在做的是:
adapter.Update
, I check that thing for the first rows and seems that it works, I use dataRowToProcess.AcceptChanges()
to achieve that).adapter.Update
被忽略,我检查了第一行的内容,似乎它有效,我使用dataRowToProcess.AcceptChanges()
来实现它)。 Launch the following command to apply the changes (More than 5 hours for 900.000 changes):启动以下命令以应用更改(900.000 次更改超过 5 小时):
cmdSQL = New SqlCommand(superQuery, cn) Dim adapter As SqlDataAdapter = New SqlDataAdapter(cmdSQL) adapter.MissingSchemaAction = MissingSchemaAction.AddWithKey Dim build As SqlCommandBuilder = New SqlCommandBuilder(adapter) build.SetAllValues = False adapter.Update(dataTableCustomersDetail) 'Insert/Update records
If I have many inserts the process, it is slower than the same amount of updates.如果我有很多插入过程,它比相同数量的更新慢。 What am I doing wrong?
我究竟做错了什么? Am I missing some
SqlDataAdapter
option?我是否缺少一些
SqlDataAdapter
选项?
Thanks谢谢
Thanks to @matsnow i figuredOut a solution with SqlBulkCopy.感谢@matsnow,我想出了一个使用SqlBulkCopy 的解决方案。 Considering that half of the table change everitime and that is a static anag i decide that a Delete/Insert of the data is the fastest way to follow (Now it takes 5-6 Minutes instead of 10).
考虑到表的一半每次都在更改并且这是一个静态分析,我决定删除/插入数据是最快的方法(现在需要 5-6 分钟而不是 10 分钟)。
Code:代码:
'Delete all table content
Dim cmd As SqlCommand = New SqlCommand("TRUNCATE TABLE " + tableName, cn)
cmd.ExecuteNonQuery()
'Insert all records
Using sbc As SqlBulkCopy = New SqlBulkCopy(cn)
sbc.DestinationTableName = tableName
sbc.BulkCopyTimeout = 1000
For Each column As DataColumn In dataTableCustomersDetail.Columns
sbc.ColumnMappings.Add(column.ToString(), column.ToString())
Next
sbc.WriteToServer(dataTableCustomersDetail)
End Using
Use Connection.BeginTransaction() to speed up the DataAdapter update.使用 Connection.BeginTransaction() 加速 DataAdapter 更新。
cn.Open() 'open connection
Dim myTrans As SQLTransaction
myTrans = cn.BeginTransaction()
'Associate the transaction with the select command object of the DataAdapter
adapter.SelectCommand.Transaction = myTrans
adapter.Update(dataTableCustomersDetail) 'do the update as before
Try
myTrans.Commit()
Catch ex As Exception
myTrans.Rollback()
End Try
cn.Close()
With 8000 rows this changes the update time from over 5 minutes to 2 seconds有了 8000 行,这会将更新时间从超过 5 分钟更改为 2 秒
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.