HttpWebRequest returns empty text

Question

When I try to fetch the content of this URL http://www.yellowpages.com.au/qld/gatton/a-12000029-listing.html

using System.Net;

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(link);
request.AllowAutoRedirect = true;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream resStream = response.GetResponseStream();
StreamReader objSR;
objSR = new StreamReader(resStream, System.Text.Encoding.GetEncoding("utf-8"));
string sResponse = objSR.ReadToEnd();

I don't get any response from the server. Please help me find out why this happens.

Thanks in advance!

Answer 1

我遇到了同样的问题，原因是我以前将方法设置为HEAD而在以后的版本中则需要解析正文。

Answer 2

It may well be looking at the user agent and refusing to serve content to a client that doesn't identify itself. Try setting the UserAgent property on your request object.

Answer 3

Looks to me like that site is checking the referrer url and may be serving up empty content if an invalid referrer is specified.

Try setting request.Referer = "http://www.google.com"; . Experiment with the referrer to see if that changes the response. I'd also try the UserAgent property as Matthew suggested.