异步HttpClient请求变慢

Question

I have list of 10 000 000 urls in text file. Now I open every of them in my await/async method - at the beging the speed is very good (near 10 000 urls / min) but while the program is running it's decreasing to reach 500 urls / min after ~10 hours. When I restart the program and run from begging the situation is the same - fast at beggining and then slower and slower. I'm working on Windows Server 2008 R2. Tested my code at various PC - some results. Can You tell me where is the problem?

 int finishedUrls = 0;
 IEnumerable<string> urls = File.ReadLines("urlslist.txt");
 await urls.ForEachAsync(500, async url =>
    {                        
        Uri newUri;
        if (!Uri.TryCreate(siteUrl, UriKind.Absolute, out newUri)) return false;
        _uri = newUri;
        var timeout = new CancellationTokenSource(TimeSpan.FromSeconds(30));
        string html = "";
        using(var _httpClient = new HttpClient { Timeout = TimeSpan.FromSeconds(30),MaxResponseContentBufferSize = 300000 }) {
            using(var _req = new HttpRequestMessage(HttpMethod.Get, _uri)){
                using( var _response = await _httpClient.SendAsync(_req,HttpCompletionOption.ResponseContentRead,timeout.Token).ConfigureAwait(false)) {

                        if (_response != null &&
                            (_response.StatusCode == HttpStatusCode.OK || _response.StatusCode == HttpStatusCode.NotFound))
                        {
                            using (var cancel = timeout.Token.Register(_response.Dispose))
                            {
                                var rawResponse = await _response.Content.ReadAsByteArrayAsync().ConfigureAwait(false);
                                html = Encoding.UTF8.GetString(rawResponse);
                            }
                        }
                }
            }
        }
        Interlocked.Increment(ref finishedUrls);
    });

http://blogs.msdn.com/b/pfxteam/archive/2012/03/05/10278165.aspx

Answer 1

I believe you are exhausting your io completion ports. You need to throttle your requests. If you need higher concurrency than a single box can handle, then distribute your concurrent requests across more machines. I'd suggest using TPL more managing the conncurrency. I ran into this exact same behavior doing similar things. Also, you should absolutely not be disposing your HttpClient per request. Pull that code out and use a single client.