I'm trying to crawl a couple of pages on my own site, but I'm getting a time-out webException("The operation has timed out") on my live environment but not on my test environment. The time-out does not occur on the same page twice, but randomly and often after some requests. After the first time-out, the frequency of the time-outs rises.
The requestUristring on test enviroment: http://localhost/Opgaver/Flytning/Haarde-hvidevarer/Bortkoersel-amerikaner-koeleskab-paa.aspx
The requestUristring on live enviroment: http://www.servicebyen.dk/Opgaver/Flytning/Haarde-hvidevarer/Bortkoersel-amerikaner-koeleskab-paa.aspx
var webRequest = (HttpWebRequest)WebRequest.Create(requestUriString);
webRequest.KeepAlive = false;
webRequest.Timeout = 3 * 30 * 1000;
webRequest.PreAuthenticate = false;
using (WebResponse webResponse = webRequest.GetResponse()) //ERROR OCCURS HERE
{
using (Stream responseStream = webResponse.GetResponseStream())
{
if (responseStream != null)
{
using (var reader = new StreamReader(responseStream))
{
string readToEnd = reader.ReadToEnd();
.....
}
}
}
}
我会尝试设置webRequest.ServicePoint.ConnectionLimit = 100
。
Try this (but only for your test environment).
Keep-alive : true if the request to the Internet resource should contain a Connection HTTP header with the value
webRequest.KeepAlive = true;
webRequest.Timeout = 95 * 95 * 100000;
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.