[英]Performance issue in bulk importing data in Dynamics CRM
I am importing data to Dynamics CRM using C# console application.我正在使用 C# 控制台应用程序将数据导入 Dynamics CRM。 I am using following code:
我正在使用以下代码:
public static void Main(string[] args)
{
int totalRecords = dbcon.GetDataCount();
int rowCount = totalRecords / 10;
for (int i = 1, j = 1; i <= totalRecords; i = i + rowCount, j = j + 1)
{
Task myTask = new Task(() => TestMethod(i, (rowCount * j)));
myTask.Start();
}
Task.WaitAll();
}
public static void TestMethod(int startSeqNo, int endSeqNo)
{
IOrganizationService service = getServiceProxcy();
DBConnection dbcon = new DBConnection();
DataTable dt = dbcon.GetData(startSeqNo, endSeqNo);
// Insert Commented
BulkCreate(service, dt);
}
public static void BulkCreate(IOrganizationService service, DataTable dt)
{
// Create an ExecuteMultipleRequest object.
ExecuteMultipleRequest multipleRequest = new ExecuteMultipleRequest()
{
// Assign settings that define execution behavior: continue on error, return responses.
Settings = new ExecuteMultipleSettings()
{
ContinueOnError = false,
ReturnResponses = true
},
// Create an empty organization request collection.
Requests = new OrganizationRequestCollection()
};
foreach (DataRow row in dt.Rows)
{
Entity entity = new Entity("new_dataimporttest");
entity["new_name"] = row["name"].ToString();
entity["new_telephone"] = row["telephone1"].ToString();
if (multipleRequest.Requests.Count == 1000)
{
// Execute all the requests in the request collection using a single web method call.
ExecuteMultipleResponse multipleResponse = (ExecuteMultipleResponse)service.Execute(multipleRequest);
multipleRequest.Requests.Clear();
}
CreateRequest createRequest = new CreateRequest { Target = entity };
multipleRequest.Requests.Add(createRequest);
}
// Execute all the requests in the request collection using a single web method call.
if (multipleRequest.Requests.Count > 0)
{
ExecuteMultipleResponse multipleResponse = (ExecuteMultipleResponse)service.Execute(multipleRequest);
}
}
I am using Task Parallel Library.我正在使用任务并行库。 It works fine but issue is that when following line is executed it takes time.
它工作正常,但问题是执行以下行时需要时间。
// Execute all the requests in the request collection using a single web method call.
// ExecuteMultipleResponse multipleResponse = (ExecuteMultipleResponse)service.Execute(multipleRequest);
I want to improve performance of code as I am importing large amount of data nearly 1 million records.我想提高代码的性能,因为我正在导入将近 100 万条记录的大量数据。 Currently it takes 1h 50 mins.
目前需要 1 小时 50 分钟。 How do I improve code to reduce execution time.
如何改进代码以减少执行时间。
With the ExecuteMultipleRequest
data throughput can only be enhanced in a limited way.使用
ExecuteMultipleRequest
只能以有限的方式提高数据吞吐量。 This is because the Dynamics CRM server processes the requests in it in sequential order, not parallel.这是因为 Dynamics CRM 服务器按顺序处理其中的请求,而不是并行处理。 Therefore your main gain is less roundtrips to the server.
因此,您的主要收获是减少了到服务器的往返次数。
Throughput can really be boosted when working with multiple threads.使用多线程时,吞吐量确实可以得到提升。 Every thread communicating with CRM must get its own
IOrganizationService
instance.每个与 CRM 通信的线程都必须获得自己的
IOrganizationService
实例。 By default a CRM server accepts up to 10 simultaneous connections from a client.默认情况下,CRM 服务器最多接受来自客户端的 10 个同时连接。 (This is the WCF default.)
(这是 WCF 默认值。)
In batch processes I tend to use a BlockingCollection<T>
with a Producer Consumer pattern: one thread produces the requests to be sent to the CRM server and multiple threads consume the requests by taking them off the collection and sending them to CRM.在批处理过程中,我倾向于使用具有生产者消费者模式的
BlockingCollection<T>
:一个线程生成要发送到 CRM 服务器的请求,多个线程通过将请求从集合中取出并将它们发送到 CRM 来使用这些请求。
If just data import is what intended here, try using SqlBulkCopy to write data directly to server ( Sample Code ).如果这里只是数据导入,请尝试使用 SqlBulkCopy 将数据直接写入服务器( 示例代码)。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.