使Parallel.ForEach等待工作直到插槽打開

Question

我正在使用Parallel.ForEach來處理很多項目。 問題是，我想根據打開的工作人員（插槽）的數量來確定哪些項目可以工作。 例如，如果我正在做8項並行工作，並且在任務1-4之間打開了一個插槽，那么我想為這些插槽分配簡單的工作。 插槽的下半部分將很辛苦。 這樣，我就不會把所有8個插槽都捆綁在一起進行辛苦/長時間運行的工作，輕松/快速的項目將首先運行。 我已經實現了如下：

編碼

const int workers = 8;
List<Thing> thingsToDo = ...; //Get the things that need to be done.
Thing[] currentlyWorkingThings = new Thing[workers]; //One slot for each worker.

void Run() {
    Parallel.ForEach(PrioritizeThings(thingsToDo), o => {
        int index = 0;

        //"PrioritizeTasks" added this thing to the list of currentlyWorkingThings.
        //Find my position in this list.
        lock (currentlyWorkingThings)
            index = currentlyWorkingThings.IndexOf(o);

        //Do work on this thing...

        //Then remove it from the list of currently working things, thereby
        //  opening a new slot when this worker returns/finishes.
        lock (currentlyWorkingThings)
            currentlyWorkingThings[index] = null;
    });
}

IEnumerable<Thing> PrioritizeThings(List<Thing> thingsToDo) {
    int slots = workers;
    int halfSlots = (int)Math.Ceiling(slots / 2f);

    //Sort thingsToDo by their difficulty, easiest first.

    //Loop until we've worked every Thing.
    while (thingsToDo.Count > 0) {
        int slotToFill = ...; //Find the first open slot.
        Thing nextThing = null;

        lock (currentlyWorkingThings) {
            //If the slot is in the "top half", get the next easy thing - otherwise
            //  get the next hard thing.
            if (slotToFill < halfSlots)
                nextThing = thingsToDo.First();
            else
                nextThing = thingsToDo.Last();

            //Add the nextThing to the list of currentlyWorkingThings and remove it from
            //  the list of thingsToDo.
            currentlyWorkingThings[slotToFill] = nextThing;
            thingsToDo.Remove(nextThing);
        }

        //Return the nextThing to work.
        yield return nextThing;
    }
}

問題

因此，我在這里看到的問題是，在打開插槽之前（在完成現有操作之前）， Parallel正在從PrioritizeThings請求進行下一步操作。 我認為Parallel很有前瞻性，並且可以使事情提前做好准備。 我希望它不執行此操作，僅在完全完成時填充一個工作程序/插槽。 我想解決此問題的唯一方法是在PrioritizeThings添加一個sleep / wait循環，直到看到合法的開放插槽，它才會使事情恢復正常工作。 但是我不喜歡這樣，我希望有某種方法可以使Parallel等待更長的時間才能開始工作。 有什么建議么？

Answer 1

有一種內置的（kinda）方法可以准確地支持您所描述的情況。

創建ForEach您需要使用非標准TaskScheduler傳遞ParallelOptions 。 困難的部分是創建一個TaskSchedueler來為您執行優先級系統，所幸微軟發布了一個示例包，其中包含一個名為“ ParallelExtensionsExtras ”的調度程序及其調度程序QueuedTaskScheduler

private static void Main(string[] args)
{
    int totalMaxConcurrancy = Environment.ProcessorCount;
    int highPriorityMaxConcurrancy = totalMaxConcurrancy / 2;

    if (highPriorityMaxConcurrancy == 0)
        highPriorityMaxConcurrancy = 1;

    QueuedTaskScheduler qts = new QueuedTaskScheduler(TaskScheduler.Default, totalMaxConcurrancy);
    var highPriortiyScheduler = qts.ActivateNewQueue(0);
    var lowPriorityScheduler = qts.ActivateNewQueue(1);

    BlockingCollection<Foo> highPriorityWork = new BlockingCollection<Foo>();
    BlockingCollection<Foo> lowPriorityWork = new BlockingCollection<Foo>();

    List<Task> processors = new List<Task>(2);

    processors.Add(Task.Factory.StartNew(() =>
    {
        Parallel.ForEach(highPriorityWork.GetConsumingPartitioner(),  //.GetConsumingPartitioner() is also from ParallelExtensionExtras, it gives better performance than .GetConsumingEnumerable() with Parallel.ForEeach(
                         new ParallelOptions() { TaskScheduler = highPriortiyScheduler, MaxDegreeOfParallelism = highPriorityMaxConcurrancy }, 
                         ProcessWork);
    }, TaskCreationOptions.LongRunning));

    processors.Add(Task.Factory.StartNew(() =>
    {
        Parallel.ForEach(lowPriorityWork.GetConsumingPartitioner(), 
                         new ParallelOptions() { TaskScheduler = lowPriorityScheduler}, 
                         ProcessWork);
    }, TaskCreationOptions.LongRunning));


    //Add some work to do here to the highPriorityWork or lowPriorityWork collections


    //Lets the blocking collections know we are no-longer going to be adding new items so it will break out of the `ForEach` once it has finished the pending work.
    highPriorityWork.CompleteAdding();
    lowPriorityWork.CompleteAdding();

    //Waits for the two collections to compleatly empty before continueing
    Task.WaitAll(processors.ToArray());
}

private static void ProcessWork(Foo work)
{
    //...
}

即使您有兩個Parallel.ForEach實例，它們運行的​​總和將不會超過您為QueuedTaskScheduler構造函數傳遞給MaxConcurrency的值，並且如果有工作，它將優先選擇先清空highPriorityWork集合在這兩個方面都可以做到（最多限制為所有可用插槽的1/2，這樣就不會阻塞低優先級隊列，您可以根據性能需要輕松地將其調整為更高或更低的比率）。

如果您不希望高優先級總是獲勝，而是希望有一個“輪循式”樣式的調度程序在兩個列表之間切換（因此，您不希望快速項目總是獲勝，而只是將它們洗牌對於速度較慢的項目），您可以將相同的優先級設置為兩個或多個隊列（或使用功能相同的RoundRobinTaskSchedulerQueue ）