Parallel.ForEach Behavior

Question

I understand that with a Parallel.ForEach loop each thread may be executing a different part of the loop at any given time. However, does each thread execute the code in the loop sequentially? I was just reading " Parallel Loops " on MSDN and it states:

Sometimes, two steps take place in the opposite order than they would if the loop were sequential. The only guarantee is that all of the loop's iterations will have run by the time the loop finishes.

Say I have the following:

IEnumerable<MyObject> myEnumerable = ...
Parallel.ForEach(myEnumerable, obj =>
{
     A();
     B();
     C();
});

I know that thread 1 may be doing A() while thread 2 might be doing C() , but will each thread execute the code sequentially in the loop. Does Thread 1 do A() B() C() or could it possibly do B() , C() , A() ?

Answer 1

An iteration is a single execution of the whole loop body, exactly as it is expressed in your code.

The iterations can (and most probably will) start, run and finish in any order (which will depend on implementation details and runtime data, like what workers become available and when). But still, each individual iteration will be executed just as if the loop was sequential (like a plain foreach ) instead of parallel.

In other words, it couldn't possibly do B() then C() then A() for any one item. It will always do A() then B() then C() for each item. You simply can't know which item will be processed in what order.

Answer 2

Parallel.ForEach will execute specified delegate for every element from myEnumerable . For example: if myEnumerable iterates through 5 elements, sequence A(), B(), C() will be called 5 times (in most cases in parallel threads but you cannot rely on this). It is not guaranteed that all threads will execute A() method for all enum elements at the same time and start executing B() methods only when all A() methods in all threads are finished (additional synchronization is needed to achieve this behaviour).

Answer 3

As is stated in the comments to your question as well, the Parallel.ForEach will execute your code in the body sequentially, as written by you.

However, the warning you read on the msdn refers to the following scenario:

Say, you have two objects, a and b , in the collection on which you iterate parallel. You must make sure that for each object you iterate over, the steps in the body are independent from any other of the iterated objects.

If they are not, values may not be what is expected from the business rules at the end. What is worse, you may never know it , as there might be no errors (if no NullException or division by zero or any other error occurs).