Dynamics CRM Web API Bulk

Question

All,

I have a C# program that is able to send data to a CRM system (by using the Web API through HTTP Post requests). However, amount of requests that I have to perform is in the order of 10000's on daily basis which as expected takes a huge amount of time to complete. I've been looking at a "bulk" functionality where I would be able to send a large amount of data at once (say 10000 task objects at once). For one reason or another I can't find such functionality being implemented, the closest I've gotten is "batch requests" though by the example provided it looks like multiple requests wrapped in a single http post request, which, i am not quite sure would be the best solution for having generated 10000 post requests wrapped in a single such one unless this is the only way to do it

So, is there a bulk functionality or is the batch requests the best I have to deal with?

Answer 1

You are correct, the batching won't help you much since all of your requests will still be processed sequentially. For that volume of data, you need to implement concurrency/parallelism/multi-threading. The integration guy on my project has been able to achieve about 100 operations per second on average using what is essentially a producer/consumer pattern. He uses an auto-growing pool of CRM connections and as soon as one request is finished processing, the connection is returned to the pool.

The Microsoft Premier Field Engineering team publishes the PFE Core Library for Dynamics CRM on CodePlex. The library has a ParallelOrganizationServiceProxy class that should do a lot of the complicated concurrency work for you. Here's an example:

public void ParallelUpdate(List<Entity> targets)
{
    try
    {
        this.Manager.ParallelProxy.Update(targets);
    }
    catch (AggregateException ae)
    {
        // Handle exceptions
    }
}