TPL Parallel.Foreach具有IO和计算密集型任务

Question

I have billions of xml log files on Azure blob storage to be processed, queried and results storage. I am using Parallel.Foreach as processing of files is independent of each other.

Parallel.ForEach<String> (listOfFeatureFiles, file => { 
  //For each file that was created
  string fileName = file;
  string directoryPath = outputfolderPath + "/" + FeatureFolderName;
  string finalFilePath = directoryPath + "/" + fileName;

  DownloadContent();
  XMLParseAndQueryData();
  UploadResultToQueue();
  DeleteLocalCopy();
});

If it were only compute intensive then i might had maximum CPU usage however with my scenario 20% of files are much bigger (in GBs) as compared to rest 80% of files. This usually results in only 50% CPU usage with 4 cores. How can i optimize it to make maximum CPU usage ie > 90% ?

My assumption is that once a task is downloading big files, no cpu is used however no new thread is created in the meantime as well which could make use of processing power. I might be wrong about this assumption and will appreciate a concrete link to its negation.

Answer 1

I built a similar application for one of my customers that also processes lots of xml files with varying sizes. The downloading will interfere with CPU usage, you can't help that. But you might optimize CPU usage by using a BlockingCollection with multiple consumers and always keep processing smaller files while a larger file is being downloaded.

Answer 2

My assumption is that once a task is downloading big files, no cpu is used however no new thread is created in the meantime as well which could make use of processing power.

Are you sure you have enough network bandwidth and that downloading the files is not actually the bottleneck of of this process?

If you are, and the slow adding of threads is actually what is slowing you down, then the quick and dirty solution would be to force the ThreadPool (which is used by Parallel.ForEach() internally) to have more threads. You can do that by calling ThreadPool.SetMinThreads .

The proper solution would be to make the IO-bound methods asynchronous and schedule them independently of the CPU-bound methods. To help with scheduling, you can use TPL Dataflow ( EnsureOrdered requires a prerelease version):

var cpuBoundOptions = new ExecutionDataflowBlockOptions
{
    MaxDegreeOfParallelism = Environment.ProcessorCount,
    EnsureOrdered = false
};

var ioBoundOptions = new ExecutionDataflowBlockOptions
{
    MaxDegreeOfParallelism = 10, // TODO: tweak this value as necessary
    EnsureOrdered = false
};

var downloadBlock = new TransformBlock<string, string>(async file =>
{
    await DownloadContentAsync(file);
    return file;
}, ioBoundOptions);

var parseBlock = new TransformBlock<string, string>(file =>
{
    XMLParseAndQueryData(file);
    return file;
}, cpuBoundOptions);

var uploadBlock = new TransformBlock<string, string>(async file =>
{
    await UploadResultToQueue(file);
    return file;
}, ioBoundOptions);

var deleteBlock = new ActionBlock<string>(file => DeleteLocalCopy(file));

var linkOptions = new DataflowLinkOptions { PropagateCompletion = true };

downloadBlock.LinkTo(parseBlock, linkOptions);
parseBlock.LinkTo(uploadBlock, linkOptions);
uploadBlock.LinkTo(deleteBlock, linkOptions);

foreach (var file in listOfFeatureFiles)
{
    downloadBlock.Post(file);
}

downloadBlock.Complete();
await deleteBlock.Completion;