在while循环中实现Parallel.Foreach

Question

I have scenario where I need to run a Parallel.Foreach within a while loop. I need to understand the impact of this implementation in terms of how the processing will take place. I will have an implementation something like this

 ConcurrentQueue<MyTable> queue = new ConcurrentQueue<MyTable>();

Here, I have initially added lot of items in queue but while execution also, more items can be added in the queue.

while(true)
{
    Parallel.Foreach(queue, (myTable) => {some processing});
    Sleep(sometime);
}

Each time one item will be de-queued and new thread will be spawned to work with it, in the meanwhile new items will be added for that I need to keep an infinite while loop.

Now, I need to understand that as concurrent queue is thread safe, I think each item will be processed one time only in spite of while above foreach but I am not sure about is that there will be multiple threads of foreach itself that will be spawning child threads or single copy of foreach will be running within while loop. I do not know how foreach itself is implemented.

Answer 1

I have scenario where I need to run a Parallel.Foreach within a while loop.

I don't think you do. You want to process new items as they come in in parallel, but I think this is not the best way to do that.

I think the best way is to use ActionBlock from TPL Dataflow. It won't waste CPU or threads when there are no items to process and if you set its MaxDegreeOfParallelism , it will process items in parallel:

ActionBlock<MyTable> actionBlock = new ActionBlock<MyTable>(
    myTable => /* some processing */,
    new ExecutionDataflowBlockOptions
    {
        MaxDegreeOfParallelism = DataflowBlockOptions.Unbounded
    });


...

actionBlock.Post(someTable);

If you don't want to or can't (it's .Net 4.5 only) use TPL Dataflow, another option would be use a single Parallel.Foreach() (no while ) together with BlockingCollection and GetConsumingPartitioner() ( not GetConsumingEnumerable() !).

Using this, the Parallel.Foreach() threads will be blocked when there are no items to process, but there also won't be any delays in processing (like the ones caused by your Sleep() ):

BlockingCollection<MyTable> queue = new BlockingCollection<MyTable>();

...

Parallel.ForEach(
    queue.GetConsumingPartitioner(), myTable => /* some processing */);

...

queue.Add(someTable);

---

I think each item will be processed one time only in spite of while above foreach but I am not sure

That's one reason why you should use one of the options above, since they mean you don't need to know much about the details of how they work, they just work.