如何在 C# 中使用多線程進行批處理

Question

我正在使用並行 foreach 在阻塞集合中添加值，但是當阻塞集合有 10 個值時，我需要對其進行一些處理，然后再次清除該阻塞集合，然后再次開始向阻塞集合添加值。

這里有兩個問題

1. 雖然我正在做一些處理，它將繼續向阻塞集合添加值，我可以在列表上加一個鎖，但當它到達鎖時，值會增加。

2. 如果我放置的鎖完全破壞了並行編程的使用，我希望在該列表中添加 object 直到這 10 條消息被處理。 我可以在這里復制列表內容並再次清空列表，同樣的問題我不能只復制 10 個項目，因為內容已經更改。

有時 if 條件永遠不會滿足，因為在檢查條件之前，值會增加。

有什么解決辦法嗎？

public static BlockingCollection<string> failedMessages = new BlockingCollection<string>();
static void Main(string[] args)
{
    var myCollection = new List<string>();
    myCollection.Add("test");
    //Consider myCollection having more than 100 items 
    Parallel.ForEach(myCollection, item =>
    {
        failedMessages.Add(item);
        if (failedMessages.Count == 10)
        {
            DoSomething();
        }
    });

}

static public void DoSomething()
{
    //dosome operation with failedMessages 
    failedMessages = new BlockingCollection<string>();
}   

    

Answer 1

這看起來像是 DataFlow 的工作：

使用批大小為 10 的BatchBlock<string>和ActionBlock<string[]>來消耗批次的示例：

using System;
using System.Threading.Tasks;
using System.Threading.Tasks.Dataflow;
                    
public class Program
{
    public static void Main()
    {
        Console.WriteLine("Hello World");
        // Set up DataFlow Blocks
        BatchBlock<string> batcher = new BatchBlock<string>( 10 );
        ActionBlock<string[]> consumer = 
            new ActionBlock<string[]>( 
                (msgs) => Console.WriteLine("Processed {0} messages.", msgs.Length)
            );
        // put them together
        batcher.LinkTo( consumer );
        
        // start posting
        Parallel.For( 0, 103, (i) => batcher.Post(string.Format("Test {0}",i)));
        
        // shutdown
        batcher.Complete();
        batcher.Completion.Wait();
    }
}

在行動： https://dotnetfiddle.net/Y9Ezg4

進一步閱讀： https://docs.microsoft.com/en-us/dotnet/standard/parallel-programming/walkthrough-using-batchblock-and-batchedjoinblock-to-improve-efficiency

---

編輯：根據要求 - 如果您不能或不想使用 DataFlow，您當然可以做類似的事情：

using System;
using System.Threading;
using System.Threading.Tasks;
using System.Collections.Generic;
using System.Collections.Concurrent;
using System.Linq;
                    
public class Program
{
    public static void Main()
    {
        FailedMessageHandler fmh = new FailedMessageHandler( new Progress<string[]>((list) => { Console.WriteLine("Handling {0} messages. [{1}]", list.Length, string.Join(",", list));}));
        Parallel.For(0,52, (i) => {fmh.Add(string.Format("Test {0,3}",i));});
        Thread.Sleep(1500); // Demo: Timeout
        var result = Parallel.For(53,107, (i) => {fmh.Add(string.Format("Test {0,3}",i));});
        while(!result.IsCompleted)
        {
            // Let Parallel.For run to end ...
            Thread.Sleep(10);
        }
        // Graceful shutdown:
        fmh.CompleteAdding();
        fmh.AwaitCompletion();
    }
}

public class FailedMessageHandler
{
    private BlockingCollection<string> workQueue = new BlockingCollection<string>();
    private List<string> currentBuffer = new List<string>(10);
    private IProgress<string[]> progress;
    private Thread workThread;
    
    public FailedMessageHandler( IProgress<string[]> progress )
    {
        this.progress = progress;
        workThread = new Thread(WatchDog);
        workThread.Start();
    }
    
    public void Add( string failedMessage )
    {
        if ( workQueue.IsAddingCompleted )
        {
            throw new InvalidOperationException("Adding is completed!");
        }
        
        workQueue.Add(failedMessage);
    }
    
    private void WatchDog()
    {
        while(true)
        {
            // Demo: Include a timeout - If there are less than 10 items
            // for x amount of time, send whatever you got so far.
            CancellationTokenSource timeout = new CancellationTokenSource(TimeSpan.FromSeconds(1));
            try{
               var failedMsg = workQueue.Take(timeout.Token);
               currentBuffer.Add(failedMsg);
               if( currentBuffer.Count >= 10 ){
                   progress.Report(currentBuffer.ToArray());
                   currentBuffer.Clear();
               }
            }
            catch(OperationCanceledException)
            {
                Console.WriteLine("TIMEOUT!");
                // timeout.
                if( currentBuffer.Any() ) // handle items if there are
                {
                    progress.Report(currentBuffer.ToArray());
                    currentBuffer.Clear();
                }
            }
            catch(InvalidOperationException)
            {
                Console.WriteLine("COMPLETED!");
                // queue has been completed.
                if( currentBuffer.Any() ) // handle remaining items
                {
                    progress.Report(currentBuffer.ToArray());
                    currentBuffer.Clear();
                }
                break;
            }
        }
        Console.WriteLine("DONE!");
    }
    
    public void CompleteAdding()
    {
        workQueue.CompleteAdding();
    }
    
    public void AwaitCompletion()
    {
        if( workThread != null )
            workThread.Join();
    }
}

在行動： https://dotnetfiddle.net/H2Rg35

請注意，使用Progress將在主線程上執行處理。 如果您改為傳遞一個Action ，它將在workThread上執行。 因此，請根據您的要求調整示例。

這也只是給出一個想法，這有很多變體，可能使用Task/Async ...