使用HttpWebRequest在C＃中获取HTTP响应时如何处理JavaScript？

Question

我有一个C＃代码（它是一个Web应用程序，托管在IIS上），在其中我使用HttpWebRequest获取HttpWebResponse。 在那里，我向任何网站发出请求并以字符串形式获取响应，然后分析响应字符串。 但是最近我得到了响应，其中页面在浏览器中加载后，JavaScript会执行数据获取。

我试图在firebug中调试它，并看到响应的底部有一个JavaScript函数，该函数会在页面加载后更新dom元素。 有什么办法可以在C＃代码中执行相同的操作。 到目前为止，我已经在网上搜索了有关此问题的解决方案。

以下是我正在使用的代码：

HttpWebResponse response = (HttpWebResponse)request.GetResponse();

        foreach (Cookie cook in response.Cookies)
        {
            Cookie cookie = new Cookie();
            cookie.Name = cook.Name;
            cookie.Value = cook.Value;
            cookie.Domain = cook.Domain;
            cookie.Expires = DateTime.Now.AddDays(10);
            cookieList.Add(cookie);
        }

        string postData = string.Format("username=" + txtUserID.Text + "&password=" + txtPwd.Text + "&url=https://example.com/&game=");
        byte[] postBytes = Encoding.UTF8.GetBytes(postData);
        HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://login.example.com/Login/authenticate");
        req.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0";
        req.KeepAlive = true;
        req.AutomaticDecompression = DecompressionMethods.GZip;

        ////set the cookie
        req.CookieContainer = new CookieContainer();
        foreach (Cookie cook in cookieList)
        {
            Cookie cookie = new Cookie();
            cookie.Name = cook.Name;
            cookie.Value = cook.Value;
            cookie.Domain = cook.Domain;
            cookie.Expires = DateTime.Now.AddDays(10);
            req.CookieContainer.Add(cookie);
        }

        req.Headers.Add("Accept-Encoding", "gzip, deflate");
        req.Headers.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6");//en-GB,en-US;q=0.8,en;q=0.6
        req.Method = "POST";
        req.Host = "login.example.com";
        req.Referer = "https://login.example.com/Login/logout";
        req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";

        req.ContentType = "application/x-www-form-urlencoded;";
        req.ContentLength = postBytes.Length;

        //getting the request stream and posting data
        StreamWriter requestwriter = new StreamWriter(req.GetRequestStream(), System.Text.Encoding.ASCII);
        requestwriter.Write(postData);
        requestwriter.Close();
        HttpWebResponse myHttpWebResponse = (HttpWebResponse)req.GetResponse();
        Stream responseStream = myHttpWebResponse.GetResponseStream();
        StreamReader myStreamReader = new StreamReader(responseStream, Encoding.ASCII);
        string responseString = myStreamReader.ReadToEnd();
        myStreamReader.Close();
        responseStream.Close();
        myHttpWebResponse.Close();

Answer 1

我终于得到了满足我需求的简便解决方案。 以下是我关注的链接： 链接至教程

以下是将获得结果的代码：

首先，您需要导入以下内容：

using System.Drawing;
using OpenQA.Selenium;
using OpenQA.Selenium.PhantomJS;
using System.Text.RegularExpressions;
using System.IO;
using HtmlAgilityPack;

现在的代码：

        var options = new PhantomJSOptions();
        var driver = new PhantomJSDriver(options);
        driver.Manage().Window.Size = new Size(1360, 728);
        var size = driver.Manage().Window.Size;

        driver.Navigate().GoToUrl("https://example.com/");
        string url = driver.Url;
        //the driver can now provide you with what you need (it will execute the script)
        //get the source of the page
        var source = driver.PageSource;
        //fully navigate the dom
        var pathElement1 = driver.FindElementByName("username");
        var pathElement2 = driver.FindElementByName("password");
        var pathElement3 = driver.FindElementByXPath("//button[@class='SubmitButton']");

        pathElement1.Clear();
        pathElement1.SendKeys("username");
        pathElement2.Clear();
        pathElement2.SendKeys("password");
        pathElement3.Click();

        //Now get the response after login button click
        source = driver.PageSource;