Killing a HttpWebRequest on slow response

Question

Occasionally while crawling, I get a VERY slow response.

It is not slow to connect, so timeout doesn't work. Its the host speed going 3-4 Bytes/second.

Is there a way to kill this request if it takes more than 10 seconds, like a timeout?

Answer 1

What you want to do is start your crawl process on its own thread that times out after a specified time. Have a look at BeginGetResponse() on the HttpWebRequest class, which starts a thread and calls a specified function when it completes. You can add a timeout mechanism to this method, as shown under the Example section on the following page: http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.begingetresponse(v=VS.71).aspx .

Sorry, I don't have any code handy to show how this is done, but the example on that page appears complete.

Answer 2

If your HttpWebRequest is in a thread you schould be able to kill the request by calling Abort on the thread itself... I know this is perhaps "overkill"... perhaps someone else has better idea...