C＃中的生產者使用者，具有多個（並行）使用者，沒有TPL數據流

Question

我正在嘗試與多個或並行的消費者實施生產者/消費者模式。

我做了一個實現，但是我想知道它有多好。 有人可以做得更好嗎？ 你們中的任何人都能發現任何錯誤嗎？

不幸的是，我不能使用TPL數據流，因為我們處在項目的最后，在程序包中放入一個額外的庫會占用大量的文檔，而且我們沒有時間。

我正在嘗試做的是加速以下部分：

anIntermediaryList = StepOne(anInputList); // I will put StepOne as Producer :-) Step one is remote call.

aResultList = StepTwo(anIntermediaryList); // I will put StepTwo as Consumer, however he also produces result. Step two is also a remote call.
// StepOne is way faster than StepTwo.

為此，我想到了將輸入列表（anInputList）分塊的想法。

StepOne將位於生產者內部，並將中間塊放入隊列中。 將有多個生產者，他們將獲得中間結果並通過StepTwo處理。

這是稍后實現的簡化版本：

Task.Run(() => {
     aChunkinputList = Split(anInputList)
     foreach(aChunk in aChunkinputList)
     {
          anIntermediaryResult = StepOne(aChunk)
          intermediaryQueue.Add(anIntermediaryResult)
     }
})

while(intermediaryQueue.HasItems)
{
     anItermediaryResult = intermediaryQueue.Dequeue()
     Task.Run(() => {
         aResultList = StepTwo(anItermediaryResult);
         resultQueue.Add(aResultList)
     }
}

我還認為並行運行的Consumers的最佳數字是：“ Environment.ProcessorCount / 2”。 我想知道這是否也是一個好主意。

現在這是我的模擬實現，問題是有人可以做得更好還是發現任何錯誤？

class Example
{
    protected static readonly int ParameterCount_ = 1000;
    protected static readonly int ChunkSize_ = 100;
    // This might be a good number for the parallel consumers.
    protected static readonly int ConsumerCount_ = Environment.ProcessorCount / 2;
    protected Semaphore mySemaphore_ = new Semaphore(Example.ConsumerCount_, Example.ConsumerCount_);

    protected ConcurrentQueue<List<int>> myIntermediaryQueue_ = new ConcurrentQueue<List<int>>();
    protected ConcurrentQueue<List<int>> myResultQueue_ = new ConcurrentQueue<List<int>>();

    public void Main()
    {
        List<int> aListToProcess = new List<int>(Example.ParameterCount_ + 1);
        aListToProcess.AddRange(Enumerable.Range(0, Example.ParameterCount_));

        Task aProducerTask = Task.Run(() => Producer(aListToProcess));

        List<Task> aTaskList = new List<Task>();            
        while(!aProducerTask.IsCompleted || myIntermediaryQueue_.Count > 0)
        {
            List<int> aChunkToProcess;
            if (myIntermediaryQueue_.TryDequeue(out aChunkToProcess))
            {
                mySemaphore_.WaitOne();
                aTaskList.Add(Task.Run(() => Consumer(aChunkToProcess)));
            }
        }

        Task.WaitAll(aTaskList.ToArray());

        List<int> aResultList = new List<int>();
        foreach(List<int> aChunk in myResultQueue_)
        {
            aResultList.AddRange(aChunk);
        }
        aResultList.Sort();
        if (aListToProcess.SequenceEqual(aResultList))
        {
            Console.WriteLine("All good!");
        }
        else
        {
            Console.WriteLine("Bad, very bad!");
        }
    }

    protected void Producer(List<int> elements_in)
    {
        List<List<int>> aChunkList = Example.SplitList(elements_in, Example.ChunkSize_);

        foreach(List<int> aChunk in aChunkList)
        {
            Console.WriteLine("Thread Id: {0} Producing from: ({1}-{2})", 
                Thread.CurrentThread.ManagedThreadId,
                aChunk.First(),
                aChunk.Last());

            myIntermediaryQueue_.Enqueue(ProduceItemsRemoteCall(aChunk));
        }
    }

    protected void Consumer(List<int> elements_in)
    {
        Console.WriteLine("Thread Id: {0} Consuming from: ({1}-{2})",
                Thread.CurrentThread.ManagedThreadId,
                Convert.ToInt32(Math.Sqrt(elements_in.First())),
                Convert.ToInt32(Math.Sqrt(elements_in.Last())));

        myResultQueue_.Enqueue(ConsumeItemsRemoteCall(elements_in));
        mySemaphore_.Release();
    }

    // Dummy Remote Call
    protected List<int> ProduceItemsRemoteCall(List<int> elements_in)
    {
        return elements_in.Select(x => x * x).ToList();
    }

    // Dummy Remote Call
    protected List<int> ConsumeItemsRemoteCall(List<int> elements_in)
    {
        return elements_in.Select(x => Convert.ToInt32(Math.Sqrt(x))).ToList();
    }

    public static List<List<int>> SplitList(List<int> masterList_in, int chunkSize_in)
    {
        List<List<int>> aReturnList = new List<List<int>>();
        for (int i = 0; i < masterList_in.Count; i += chunkSize_in)
        {
            aReturnList.Add(masterList_in.GetRange(i, Math.Min(chunkSize_in, masterList_in.Count - i)));
        }
        return aReturnList;
    }
}

主功能：

class Program
{
    static void Main(string[] args)
    {
        Example anExample = new Example();
        anExample.Main();
    }
}

再見拉斯洛

Answer 1

基於這些評論，我發布了第二版和第三版： https : //codereview.stackexchange.com/questions/71182/producer-consumer-in-c-with-multiple-parallel-consumers-and-no-tpl-數據流/ 71233＃71233