Parallel.Foreach 与所有并行进程共享绝对索引

Question

I'm running a process which, for example, works on 10 files at a time. I need to assign a serial number based on the input order of the input file array. So, for each parallel process, I need to assign the serial numbers to be used in the parallel process in the same order as the input string array myFiles . Do I need some type of threadsafe or concurrent int? What's the correct approach?

var results = new ConcurrentQueue<string>();
var options = new ParallelOptions
    { MaxDegreeOfParallelism = Environment.ProcessorCount * 10 };
int startSerialNumber = 1;
if (runParallel)
{
    Parallel.ForEach(myFiles, options, (myFile) =>
    {
        var newMyFile = WorkOnMyFile(myFile,startSerialNumber);
        startSerialNumber += SubFileCount; // <--This needs to be shared
            // for all parallel processes where how do I control incrementing?
        results.Enqueue(RunExeTask(newMyFile, outputDirectory,false));
    });
}

Answer 1

Generate the serial numbers outside of the parallel processing. Incrementing a number is trivial so it's not like you need to do it on multiple threads. As you generate them, pair them with the items in your list to create a new list containing both, then iterate over that.

var myData = myFiles
    .Select
    (
        (f, i) => new { File = f, SerialNumber = startingSerialNuber + (i * SubFileCount) }
    )
    .ToList();
Parallel.ForEach(myData, options, (myItem) =>
{
    myFile = myItem.File;
    serialNumber = myItem.SerialNumber;
    var newMyFile = WorkOnMyFile(myFile,serialNumber);
    results.Enqueue(RunExeTask(newMyFile, outputDirectory,false));
});

Answer 2

I suggest to use PLINQ instead of the Parallel class, because the former is inherently able to collect thread-safely the processed results, and return them (optionally) in their original order. It also makes easy to get the index of the currently processed item, by using the Select overload that accepts an index:

public static ParallelQuery<TResult> Select<TSource, TResult> (
    this ParallelQuery<TSource> source,
    Func<TSource, int, TResult> selector);

Usage example:

string[] results = myFiles
    .AsParallel()
    .AsOrdered() // Optional, by default the original order will not be preserved
    .WithDegreeOfParallelism(runParallel ? Environment.ProcessorCount : 1)
    .Select((myFile, index) =>
    {
        var newMyFile = WorkOnMyFile(myFile, index);
        return RunExeTask(newMyFile, outputDirectory, false);
    }).ToArray();

Answer 3

You could try something like this:

    var results = new ConcurrentQueue<string>();
    var options = new ParallelOptions { MaxDegreeOfParallelism = Environment.ProcessorCount * 10 };
    int startSerialNumber = 1;
    if (runParallel)
    {
        new Thread(() =>
        {
            Task.Run(() =>
            {
                Parallel.ForEach(myFiles, options, (myFile) =>
                {
                    var newMyFile = WorkOnMyFile(myFile, startSerialNumber);
                    Interlocked.Add(ref startSerialNumber, SubFileCount);
                results.Enqueue(RunExeTask(newMyFile, outputDirectory, false));
                });
            }).Wait();
        }).Start();
    }

Essentially this runs your operation on a background thread that waits for a task to complete before carrying on. As long as you don't reference startSerialNumber anywhere else you should be fine.