使用任务并行库

Question

Im just starting out writing and using the TPL in .NET 4.5 and was wondering if you could help me.

Basically, my WPF program performs heavy duty work over a set of database records which could be in the range of 5 to 10 million. I want to make use of the TPL library to efficiently manage this work while keeping the UI responsive.

My scenario is as follows, I was planning on using 3 separate tasks to retrieve data of say 3 million each, and feed them to my "worker".

And I want a separate task that takes each item from ALL the items from the above tasks and does some work with it.

So i want a global List of items that all 3 tasks can put items in, my fourth task, should then pick items from this master list one at a time and do some work with it, it should wait if the list is empty for it to be filled up again etc.

Could you please provide some guidance on how to do this with TPL? I am a newbie with the TPL. Some sample code would be good.

Answer 1

Consider storing your items from each Task in a global, thread-safe list object, like a ConcurrentStack or ConcurrentQueue .

For more info, see: http://msdn.microsoft.com/en-us/library/system.collections.concurrent(v=vs.110).aspx

Answer 2

If I understand the problem description correctly, you have a producer/consumer scenario where there are three producers, each retrieving independent records from a database, and a single consumer.

If this is the case, then it seems to me that you want to start with BlockingCollection<T> , where T is the type of object the producers are retrieving from the database. The underlying collection to use depends on how you want to process the data. Most common scenarios are FIFO and unordered, represented by ConcurrentQueue<T> and ConcurrentBag<T> , respectively.

Here's a simple code example that demonstrates how that would work:

BlockingCollection<int> consumeFrom = new BlockingCollection<int>();
int producerCount = 3;

for (int i = 0; i < producerCount; i++)
{
    int taskValue = i;

    // Make dummy task for example
    Task.Run(() =>
    {
        for (int j = 0; j < 10; j++)
        {
            Thread.Sleep(1000);
            consumeFrom.Add(taskValue * 10 + j);
        }

        if (Interlocked.Decrement(ref producerCount) == 0)
        {
            consumeFrom.CompleteAdding();
        }
    });
}

foreach (int i in consumeFrom.GetConsumingEnumerable())
{
    Console.WriteLine(i);
}

Note that this uses the default underlying data source of ConcurrentQueue<T> . You can provide other implementations of IProducerConsumerCollection<T> to the BlockingCollection<T> constructor, eg:

BlockingCollection<int> consumeFrom =
    new BlockingCollection<int>(new ConcurrentBag<int>());