![](/img/trans.png)
[英]How to limit the number of active Tasks running via the Parallel Task Library?
[英]Limit number of Threads in Task Parallel Library
我有幾百個文件需要上傳到Azure Blob存儲。
我想使用並行任務庫。
但是,如何在foreach文件列表中運行所有100個線程上傳,我如何限制它可以使用的最大線程數並並行完成作業。 還是自動平衡事物?
你根本不應該使用線程。 有一個基於Task
的API,它自然是異步的: CloudBlockBlob.UploadFromFileAsync 。 與async/await
和SemaphoreSlim
一起使用它可以限制並行上傳的次數。
示例(未經測試):
const MAX_PARALLEL_UPLOADS = 5;
async Task UploadFiles()
{
var files = new List<string>();
// ... add files to the list
// init the blob block and
// upload files asynchronously
using (var blobBlock = new CloudBlockBlob(url, credentials))
using (var semaphore = new SemaphoreSlim(MAX_PARALLEL_UPLOADS))
{
var tasks = files.Select(async(filename) =>
{
await semaphore.WaitAsync();
try
{
await blobBlock.UploadFromFileAsync(filename, FileMode.Create);
}
finally
{
semaphore.Release();
}
}).ToArray();
await Task.WhenAll(tasks);
}
}
你嘗試過使用MaxDegreeOfParallelism嗎? 像這樣:
System.Threading.Tasks.Parallel.Invoke(
new Tasks.ParallelOptions {MaxDegreeOfParallelism = 5 }, actionsArray)
基本上,您將要為要上載的每個文件創建一個Action或Task,將它們放入List中,然后處理該列表,從而限制可以並行處理的數量。
我的博客文章展示了如何使用“任務”和“操作”執行此操作,並提供了一個示例項目,您可以下載並運行以查看兩者的實際操作。
如果使用Actions,則可以使用內置的.Net Parallel.Invoke函數。 在這里,我們將其限制為最多並行運行5個線程。
var listOfActions = new List<Action>();
foreach (var file in files)
{
var localFile = file;
// Note that we create the Task here, but do not start it.
listOfTasks.Add(new Task(() => blobBlock.UploadFromFileAsync(localFile, FileMode.Create)));
}
var options = new ParallelOptions {MaxDegreeOfParallelism = 5};
Parallel.Invoke(options, listOfActions.ToArray());
但是,此選項不使用UploadFromFileAsync的異步特性,因此您可能希望使用下面的Task示例。
使用任務,沒有內置功能。 但是,您可以使用我在博客上提供的那個。
/// <summary>
/// Starts the given tasks and waits for them to complete. This will run, at most, the specified number of tasks in parallel.
/// <para>NOTE: If one of the given tasks has already been started, an exception will be thrown.</para>
/// </summary>
/// <param name="tasksToRun">The tasks to run.</param>
/// <param name="maxTasksToRunInParallel">The maximum number of tasks to run in parallel.</param>
/// <param name="cancellationToken">The cancellation token.</param>
public static async Task StartAndWaitAllThrottledAsync(IEnumerable<Task> tasksToRun, int maxTasksToRunInParallel, CancellationToken cancellationToken = new CancellationToken())
{
await StartAndWaitAllThrottledAsync(tasksToRun, maxTasksToRunInParallel, -1, cancellationToken);
}
/// <summary>
/// Starts the given tasks and waits for them to complete. This will run the specified number of tasks in parallel.
/// <para>NOTE: If a timeout is reached before the Task completes, another Task may be started, potentially running more than the specified maximum allowed.</para>
/// <para>NOTE: If one of the given tasks has already been started, an exception will be thrown.</para>
/// </summary>
/// <param name="tasksToRun">The tasks to run.</param>
/// <param name="maxTasksToRunInParallel">The maximum number of tasks to run in parallel.</param>
/// <param name="timeoutInMilliseconds">The maximum milliseconds we should allow the max tasks to run in parallel before allowing another task to start. Specify -1 to wait indefinitely.</param>
/// <param name="cancellationToken">The cancellation token.</param>
public static async Task StartAndWaitAllThrottledAsync(IEnumerable<Task> tasksToRun, int maxTasksToRunInParallel, int timeoutInMilliseconds, CancellationToken cancellationToken = new CancellationToken())
{
// Convert to a list of tasks so that we don't enumerate over it multiple times needlessly.
var tasks = tasksToRun.ToList();
using (var throttler = new SemaphoreSlim(maxTasksToRunInParallel))
{
var postTaskTasks = new List<Task>();
// Have each task notify the throttler when it completes so that it decrements the number of tasks currently running.
tasks.ForEach(t => postTaskTasks.Add(t.ContinueWith(tsk => throttler.Release())));
// Start running each task.
foreach (var task in tasks)
{
// Increment the number of tasks currently running and wait if too many are running.
await throttler.WaitAsync(timeoutInMilliseconds, cancellationToken);
cancellationToken.ThrowIfCancellationRequested();
task.Start();
}
// Wait for all of the provided tasks to complete.
// We wait on the list of "post" tasks instead of the original tasks, otherwise there is a potential race condition where the throttler's using block is exited before some Tasks have had their "post" action completed, which references the throttler, resulting in an exception due to accessing a disposed object.
await Task.WhenAll(postTaskTasks.ToArray());
}
}
然后創建任務列表並調用函數讓它們運行,一次最多同時執行5個,你可以這樣做:
var listOfTasks = new List<Task>();
foreach (var file in files)
{
var localFile = file;
// Note that we create the Task here, but do not start it.
listOfTasks.Add(new Task(async () => await blobBlock.UploadFromFileAsync(localFile, FileMode.Create)));
}
await Tasks.StartAndWaitAllThrottledAsync(listOfTasks, 5);
你可以通過運行這個來找到:
class Program
{
static void Main(string[] args)
{
var list = new List<int>();
for (int i = 0; i < 100; i++)
{
list.Add(i);
}
var runningIndex = 0;
Task.Factory.StartNew(() => Action(ref runningIndex));
Parallel.ForEach(list, i =>
{
runningIndex ++;
Console.WriteLine(i);
Thread.Sleep(3000);
});
Console.ReadKey();
}
private static void Action(ref int number)
{
while (true)
{
Console.WriteLine("worked through {0}", number);
Thread.Sleep(2900);
}
}
}
正如您所看到的,並行性的數量在開始時較小,變得更大,並且在結束時變小。 所以肯定會有某種自動優化。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.