C# MultiThreading Loop entire DataTable while limiting threads to 4

Question

This may be a tricky question to ask, but what I have is a DataTable that contains 1000 rows. Foreach of these rows I want to process on a new thread. However I want to limit the threads to 4 threads. So basically I'm constently keeping 4 threads running until the whole datatable has been processed.

currently I have this;

  foreach (DataRow dtRow in urlTable.Rows)
        {
            for (int i = 0; i < 4; i++)
            {
                Thread thread = new Thread(() => MasterCrawlerClass.MasterCrawlBegin(dtRow));
                thread.Start();
            }
        }

I know this is backwards but i'm not sure how to achieve what I'm looking for. I thought of a very complicated while loop but maybe that's not the best way? Any help is always appreciated.

Answer 1

Simplest solution would be in case you have 4 CPU cores - Parallel LINQ +Degree of parallelism == 4 would give you one threads per CPU core, otherwise you have manually distribute records between threads/tasks, see both solutions below:

PLINQ solution:

urlTable.Rows.AsParallel().WithDegreeOfParallelism(4)
             .Select(....)

Manual distribution:

You can distribute items by worker threads manually using simple trick: N-thread would pick up each N+4 item from the input list, for instance:

• First thread: Each 0+4 == 0, 3, 7...
• Second: Each 1+4 == 1, 4, 8...
• Third: Each 2+4 ==...

Task Parallel Library solution:

private void ProcessItems(IEnumerable<string> items)
{
     // TODO: ..
}

var items = new List<string>(Enumerable.Range(0, 1000)
                                       .Select(i => i + "_ITEM"));
var items1 = items.Where((item, index) => (index + 0) % 4 == 0);
var items2 = items.Where((item, index) => (index + 1) % 4 == 0);
var items3 = items.Where((item, index) => (index + 2) % 4 == 0);
var items4 = items.Where((item, index) => (index + 3) % 4 == 0);

var tasks = new Task[]
    {
       factory.StartNew(() => ProcessItems((items1))),
       factory.StartNew(() => ProcessItems((items2))),
       factory.StartNew(() => ProcessItems((items3))),
       factory.StartNew(() => ProcessItems((items4)))
    };

Task.WaitAll(tasks);

MSDN:

• WithDegreeOfParallelism() :
• Introduction to PLINQ