使用SemaphoreSlim和Continuewith下載Urls

Question

我正在嘗試使用SemaphoreSlim和ContinueWith來限制我正在運行的並發任務的數量。 但是運行時行為與我的期望有很大不同。

我為ServicePointManager.DefaultConnectionLimit設置的值等於288，並且由於我已經初始化了SemaphoreSlim（100），因此我對運行時行為的期望是，代碼應首先生成100個線程，然后在第一個啟動時啟動新任務任務完成。

var sr =
    new StreamReader(
        @"UrlList.tsv");

var urlList = new List<string>();
for (int i = 0; i < 1000; i++)
{
    string line = sr.ReadLine();
    string[] tokens = line.Split('\t');
    string url = tokens[4];
    urlList.Add(url);
}

ServicePointManager.DefaultConnectionLimit = 12*Environment.ProcessorCount;
Console.WriteLine(DateTime.Now + "\t" + ServicePointManager.DefaultConnectionLimit);

var tasks = new Task[urlList.Count];
var semaphore = new SemaphoreSlim(100);
var client = new HttpClient();
int cnt = 0;
for (int i = 0; i < urlList.Count; i++)
{
    int i1 = i;
    tasks[i] = semaphore.WaitAsync().ContinueWith(task =>
    {
        Console.WriteLine(DateTime.Now + "\t" + ++cnt);
        var t = client.GetStringAsync(urlList[i1]);
        Console.WriteLine(t.Result);
        semaphore.Release();
        return t.Result;
    });
}
Task.WhenAll(tasks).GetAwaiter().GetResult();

輸出看起來像這樣：

4/6/2015 11:36:12 PM    288
4/6/2015 11:36:12 PM    1
4/6/2015 11:36:12 PM    7
4/6/2015 11:36:12 PM    10
4/6/2015 11:36:12 PM    11
4/6/2015 11:36:12 PM    12
4/6/2015 11:36:12 PM    3
4/6/2015 11:36:12 PM    4
4/6/2015 11:36:12 PM    2
4/6/2015 11:36:12 PM    5
4/6/2015 11:36:12 PM    8
4/6/2015 11:36:12 PM    6
4/6/2015 11:36:12 PM    9
4/6/2015 11:36:12 PM    21
4/6/2015 11:36:12 PM    17
4/6/2015 11:36:12 PM    14
4/6/2015 11:36:12 PM    15
4/6/2015 11:36:12 PM    13
4/6/2015 11:36:12 PM    22
4/6/2015 11:36:12 PM    16
4/6/2015 11:36:12 PM    23
4/6/2015 11:36:12 PM    20
4/6/2015 11:36:12 PM    19
4/6/2015 11:36:12 PM    24
4/6/2015 11:36:12 PM    25
4/6/2015 11:36:12 PM    18
4/6/2015 11:36:13 PM    26
4/6/2015 11:36:14 PM    27
4/6/2015 11:36:15 PM    28

因此，似乎未按我期望的方式生成線程，也未顯示任何Url內容。 我的代碼到底是什么問題？

Answer 1

試試這樣的東西：

async Task<IEnumerable<string>> DoItAsync(int threads, IEnumerable<string> urls)
{
    ServicePointManager.DefaultConnectionLimit = 12*Environment.ProcessorCount;
    Console.WriteLine("{0:HH:mm:ss.ffffff}\t{1}", DateTime.Now, ServicePointManager.DefaultConnectionLimit);

    var semaphore = new SemaphoreSlim(threads);
    var client = new HttpClient();
    var cnt = 0;
    var tasks = new List<Task<string>>();
    foreach (var url in urls)
    {
        tasks.Add(((Func<Task<string>>)(async () =>
            {
                await semaphore.WaitAsync();

                var c = ++cnt;
                Console.WriteLine("{0:HH:mm:ss.ffffff}\t{1}\t{2}", DateTime.Now, c, url);
                var s = await client.GetStringAsync(url);
                Console.WriteLine("{0:HH:mm:ss.ffffff}\t{1}\t{2}\t{3}", DateTime.Now, c, url, s.Substring(0, 20));
                semaphore.Release();
                return s;
            }))());
    }

    return await Task.WhenAll(tasks);
}