Performance issue in bulk importing data in Dynamics CRM

Question

I am importing data to Dynamics CRM using C# console application. I am using following code:

    public static void Main(string[] args)
    {
            int totalRecords = dbcon.GetDataCount();
            int rowCount = totalRecords / 10;

            for (int i = 1, j = 1; i <= totalRecords; i = i + rowCount, j = j + 1)
            {
                Task myTask = new Task(() => TestMethod(i, (rowCount * j)));
                myTask.Start();
            }
          Task.WaitAll();
     }


public static void TestMethod(int startSeqNo, int endSeqNo)
{            
   IOrganizationService service = getServiceProxcy();
    DBConnection dbcon = new DBConnection();
    DataTable dt = dbcon.GetData(startSeqNo, endSeqNo);
    // Insert Commented
    BulkCreate(service, dt);    
 }



public static void BulkCreate(IOrganizationService service, DataTable dt)
{
  // Create an ExecuteMultipleRequest object.
  ExecuteMultipleRequest multipleRequest = new ExecuteMultipleRequest()
  {
    // Assign settings that define execution behavior: continue on error, return responses. 
    Settings = new ExecuteMultipleSettings()
    {
        ContinueOnError = false,
        ReturnResponses = true
    },
    // Create an empty organization request collection.
    Requests = new OrganizationRequestCollection()
};

foreach (DataRow row in dt.Rows)
{
    Entity entity = new Entity("new_dataimporttest");
    entity["new_name"] = row["name"].ToString();
    entity["new_telephone"] = row["telephone1"].ToString();
    if (multipleRequest.Requests.Count == 1000)
    {
        // Execute all the requests in the request collection using a single web method call.
        ExecuteMultipleResponse multipleResponse = (ExecuteMultipleResponse)service.Execute(multipleRequest);
        multipleRequest.Requests.Clear();
    }
    CreateRequest createRequest = new CreateRequest { Target = entity };
    multipleRequest.Requests.Add(createRequest);        
}

// Execute all the requests in the request collection using a single web method call.
if (multipleRequest.Requests.Count > 0)
{
    ExecuteMultipleResponse multipleResponse = (ExecuteMultipleResponse)service.Execute(multipleRequest);
}
}

I am using Task Parallel Library. It works fine but issue is that when following line is executed it takes time.

// Execute all the requests in the request collection using a single web method call.
    //            ExecuteMultipleResponse multipleResponse = (ExecuteMultipleResponse)service.Execute(multipleRequest);

I want to improve performance of code as I am importing large amount of data nearly 1 million records. Currently it takes 1h 50 mins. How do I improve code to reduce execution time.

Answer 1

With the ExecuteMultipleRequest data throughput can only be enhanced in a limited way. This is because the Dynamics CRM server processes the requests in it in sequential order, not parallel. Therefore your main gain is less roundtrips to the server.

Throughput can really be boosted when working with multiple threads. Every thread communicating with CRM must get its own IOrganizationService instance. By default a CRM server accepts up to 10 simultaneous connections from a client. (This is the WCF default.)

In batch processes I tend to use a BlockingCollection<T> with a Producer Consumer pattern: one thread produces the requests to be sent to the CRM server and multiple threads consume the requests by taking them off the collection and sending them to CRM.

Answer 2

• You could have two methods execute multiplerequest in the same process(500 for each), using that you can cut the time by half.
• Analize in which part it is wasting more time in the execution or in the foreach create. And write here so I can't help you in a better way

Answer 3

If just data import is what intended here, try using SqlBulkCopy to write data directly to server ( Sample Code ).