[英]C# - how to do multiple web requests at the same time
我寫了一個代碼來檢查 url,但是,ir 的工作真的很慢。
我的代碼:
Parallel.ForEach(urls, new ParallelOptions {
MaxDegreeOfParallelism = 10
}, s => {
try {
using(HttpRequest httpRequest = new HttpRequest()) {
httpRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0";
httpRequest.Cookies = new CookieDictionary(false);
httpRequest.ConnectTimeout = 10000;
httpRequest.ReadWriteTimeout = 10000;
httpRequest.KeepAlive = true;
httpRequest.IgnoreProtocolErrors = true;
string check = httpRequest.Get(s + "'", null).ToString();
if (errors.Any(new Func < string, bool > (check.Contains))) {
Valid.Add(s);
Console.WriteLine(s);
File.WriteAllLines(Environment.CurrentDirectory + "/Good.txt", Valid);
}
}
} catch {
}
});
您的服務調用不太可能受CPU 限制。 因此,啟動更多線程來處理負載可能不是最好的方法——如果您使用async
和await
,您將獲得更好的吞吐量,如果可以,使用更現代的HttpClient而不是 HttpRequest 或 HttpWebRequest。
以下是如何執行此操作的示例:
var client = new HttpClient();
//Start with a list of URLs
var urls = new string[]
{
"http://www.google.com",
"http://www.bing.com"
};
//Start requests for all of them
var requests = urls.Select
(
url => client.GetAsync(url)
).ToList();
//Wait for all the requests to finish
await Task.WhenAll(requests);
//Get the responses
var responses = requests.Select
(
task => task.Result
);
foreach (var r in responses)
{
// Extract the message body
var s = await r.Content.ReadAsStringAsync();
Console.WriteLine(s);
}
嘗試做如下。
Parallel.ForEach(urls, new ParallelOptions { MaxDegreeOfParallelism = Environment.ProcessorCount - 1 }
至少它可以通過保留 1 來確保使用所有內核,這樣您的機器就不會耗盡內存。
另外,請考慮@KSib 評論。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.