[英]Bulk upload strategy for SQL server
I am uploading data from CSV to SQL table using the following function. 我正在使用以下功能将数据从CSV上传到SQL表。
Is there a better way to do it? 有更好的方法吗?
I am concerned about , right now, connection hold for long time. 我关注的是,现在,连接保持了很长时间。 Hence need to be reduced.
因此需要减少。
public bool SaveProxyBulkUploadData(List<AddServerPError> saveBulkUploadData)
{
try
{
foreach (AddServerPError addServetData in saveBulkUploadData)
{
DbCommand dbCmd = CitiScriptExecutionDB.GetStoredProcCommand("USP_HS_InsertProxyBulkUploadData");
CitiScriptExecutionDB.AddInParameter(dbCmd, "@groupid", DbType.String, addServetData.GroupId);
CitiScriptExecutionDB.AddInParameter(dbCmd, "@proxyname", DbType.String, addServetData.ProxyName);
CitiScriptExecutionDB.AddInParameter(dbCmd, "@proxytype", DbType.String, addServetData.ProxyType);
CitiScriptExecutionDB.AddInParameter(dbCmd, "@sa_spoc_dl", DbType.String, addServetData.SA_SPOC_DL);
CitiScriptExecutionDB.AddInParameter(dbCmd, "@assignmentgroup", DbType.String, addServetData.AssignmentGroup);
CitiScriptExecutionDB.AddInParameter(dbCmd, "@createdby", DbType.String, addServetData.CreatedBy);
CitiScriptExecutionDB.ExecuteNonQuery(dbCmd);
}
return true;
}
catch (Exception ex)
{
string msg = ex.Message + ex.StackTrace;
throw;
}
}
Stored Procedure 储存程序
CREATE PROCEDURE USP_HS_InsertBulkUploadData
@groupid int,
@hostname varchar(50),
@type1 varchar(50),
@type2 varchar(50),
@createdby varchar(50)
AS
BEGIN
Insert into [dbo].[EUCUsecaseGroupInputParam]
(
GroupId,
HostName,
Type1,
Type2,
CreatedBy,
CreatedDate,
UpdatedBy,
UpdatedDate,
IsActive
)
Values(
@groupId,
@hostname,
@type1,
@type2,
@createdby,
GetDate(),
null,
null,
1
)
END
Your best bet here is probably SqlBulkCopy
, which throws raw TDS at the server very efficiently. 您最好的选择可能是
SqlBulkCopy
,它可以非常有效地将原始TDS SqlBulkCopy
服务器。 SqlBulkCopy
takes two types of input: SqlBulkCopy
接受两种类型的输入:
DataTable
IDataReader
So at that point you have 3 options: 因此,此时您有3个选择:
List<AddServerPError>
to a DataTable
manually List<AddServerPError>
转换为DataTable
FastMember
to obtain an IDataReader
over your existing list (an example of this is at the bottom of the project page: https://github.com/mgravell/fast-member ) - on nuget: https://www.nuget.org/packages/FastMember/ FastMember
的方法在现有列表上获取IDataReader
(此示例在项目页面的底部: https: //github.com/mgravell/fast-member )-在nuget上: https:// www。 nuget.org/packages/FastMember/ IDataReader
- there is a "LumenWorks" CsvReader
that has always worked well for that; IDataReader
直接读取CSV-有一个“ LumenWorks” CsvReader
一直可以很好地工作; the closest nuget link I can see is https://www.nuget.org/packages/LumenWorksCsvReader/ (although this isn't the original, it has been extended) You can consider using Cinchoo ETL , an open source library to do the bulk upload CSV file to database. 您可以考虑使用Cinchoo ETL (一种开放源代码库)将CSV文件批量上传到数据库。
Option 1: 选项1:
Load the CSV file straight to database 直接将CSV文件加载到数据库
string connectionstring = @"#YOUR DB ConnectionString#";
using (SqlBulkCopy bcp = new SqlBulkCopy(connectionstring))
{
using (var p = new ChoCSVReader("#YOUR CSV FILE#"))
{
bcp.DestinationTableName = "#TABLENAME#";
bcp.EnableStreaming = true;
bcp.BatchSize = 10000;
bcp.BulkCopyTimeout = 0;
bcp.NotifyAfter = 100;
bcp.SqlRowsCopied += delegate (object sender, SqlRowsCopiedEventArgs e)
{
Console.WriteLine(e.RowsCopied.ToString("#,##0") + " rows copied.");
};
bcp.WriteToServer(p.AsDataReader());
}
}
Option 2: 选项2:
If the load of CSV done already and outputted as List<AddServerPError>
, you still can upload them to database as below 如果已经完成了CSV的加载并以
List<AddServerPError>
输出,则仍然可以按如下所示将它们上传到数据库
List<AddServerPError> objs = # Your input objects #;
string connectionstring = @"#YOUR DB ConnectionString#";
using (SqlBulkCopy bcp = new SqlBulkCopy(connectionstring))
{
bcp.DestinationTableName = "#TABLENAME#";
bcp.EnableStreaming = true;
bcp.BatchSize = 10000;
bcp.BulkCopyTimeout = 0;
bcp.NotifyAfter = 100;
bcp.SqlRowsCopied += delegate (object sender, SqlRowsCopiedEventArgs e)
{
Console.WriteLine(e.RowsCopied.ToString("#,##0") + " rows copied.");
};
bcp.WriteToServer(objs.AsDataReader());
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.