W3C工具的HTML敏捷包问题

Question

I'm trying to access the HTML result of the w3C mobileOK Checker by passing a url such as

http://validator.w3.org/mobile/check?async=false&docAddr=http%3A%2F%2Fwww.google.com/%2Ftv%2F

The URL works if you put it in a browser but I can't seem to be able to access it via the HTMLAgilityPack. The reason for this probably is that the URL needs to send a number of requests to it's server since it's an online testing, therefore it's not just a "static" URL. I have accessed other URLs without any problems. Below is my code:

HtmlAgilityPack.HtmlDocument webGet = new HtmlAgilityPack.HtmlDocument();
HtmlWeb hw = new HtmlWeb();
webGet = hw.Load("http://validator.w3.org/mobile/check?async=false&docAddr=http%3A%2F%2Fwww.google.com/%2Ftv%2F");
HtmlNodeCollection nodes = webGet.DocumentNode.SelectNodes("//head");

if (nodes != null)
{
    foreach(HtmlNode n in nodes)
    {
        string x = n.InnerHtml;
    }                        
}

Edit: I tried to access it via Stream Reader and the website returns the following error: The remote server returned an error: (403) Forbidden. I'm guessing that it's related.

Answer 1

I checked your example and was able to verify the described behaviour. It seems to me that w3.org checks if the request program is a browser or anything else .

I created an extended webClient class for another project on my own, and was able to access the given url with success.

Program.cs

WebClientExtended client = new WebClientExtended();
string exportPath = @"e:\temp"; // adapt to your own needs

string url = "http://validator.w3.org/mobile/check?async=false&docAddr=http%3A%2F%2Fwww.google.com/%2Ftv%2F";
/// load html by using cusomt webClient class
/// but use HtmlAgilityPack for parsing, manipulation aso
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(System.Text.Encoding.UTF8.GetString(client.DownloadData(url)));
doc.Save(Path.Combine(exportPath, "check.html"));

WebClientExtended

public class WebClientExtended : WebClient
{
    #region Felder
    private CookieContainer container = new CookieContainer();
    #endregion

    #region Eigenschaften
    public CookieContainer CookieContainer
    {
        get { return container; }
        set { container = value; }
    }
    #endregion

    #region Konstruktoren
    public WebClientExtended()
    {
        this.container = new CookieContainer();
    }
    #endregion

    #region Methoden
    protected override WebRequest GetWebRequest(Uri address)
    {
        WebRequest r = base.GetWebRequest(address);
        var request = r as HttpWebRequest;
        request.AllowAutoRedirect = false; 
        request.ServicePoint.Expect100Continue = false;
        if (request != null)
        {
            request.CookieContainer = container;
        }

        ((HttpWebRequest)r).Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
        ((HttpWebRequest)r).UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"; //IE

        r.Headers.Set("Accept-Encoding", "gzip, deflate, sdch");
        r.Headers.Set("Accept-Language", "de-AT,de;q=0.8,en;q=0.6,en-US;q=0.4,fr;q=0.2");
        r.Headers.Add(System.Net.HttpRequestHeader.KeepAlive, "1");

        ((HttpWebRequest)r).AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
        return r;
    }

    protected override WebResponse GetWebResponse(WebRequest request)
    {
        WebResponse response = base.GetWebResponse(request);

        if (!string.IsNullOrEmpty(response.Headers["Location"]))
        {
            request = GetWebRequest(new Uri(response.Headers["Location"]));
            request.ContentLength = 0;
            response = GetWebResponse(request);
        }

        return response;
    }
    #endregion
}

I think the crucial point is the addition/manipulation of userAgent, Accept-encoding, -language strings . The result of my code is the downloaded page check.html .