[英]HttpWebRequest with pages that have dynamically generated html from javascript?
This is kind of a two part question. 这是一个分为两个部分的问题。 Is it possible to retrieve data from a page that loads content dynamically through javascript by using WebClient/HttpWebRequest?
是否可以从使用WebClient / HttpWebRequest通过javascript动态加载内容的页面中检索数据?
I'd also like to how I would be able to use WebClient/HttpWebRequest to replicate a XMLHttpRequest like you would see executed via javascript. 我还想知道如何使用WebClient / HttpWebRequest复制XMLHttpRequest,就像通过JavaScript执行一样。
Edit: I captured the headers of the request I am trying to replicate which looks like this: http://www.tagged.com/api/?application_id=user&format=json&session_token=6thk20fhv7d727emgdhfka6034 编辑:我捕获了我要复制的请求的标头,如下所示: http : //www.tagged.com/api/?application_id=user&format=json&session_token=6thk20fhv7d727emgdhfka6034
POST /api/?application_id=user&format=json&session_token=6thk20fhv7d727emgdhfka6034 HTTP/1.1 Host: www.tagged.com POST / api /?application_id = user&format = json&session_token = 6thk20fhv7d727emgdhfka6034 HTTP / 1.1主机:www.tagged.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9, / ;q=0.8 Accept-Language: en-us,en;q=0.5 接受:text / html,application / xhtml + xml,application / xml; q = 0.9, / ; q = 0.8 Accept-Language:zh-cn,en; q = 0.5
Accept-Encoding: gzip,deflate 接受编码:gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 接受字符集:ISO-8859-1,utf-8; q = 0.7,*; q = 0.7
X-Requested-With: XMLHttpRequest X-Requested-With:XMLHttpRequest
Content-Type: application/x-www-form-urlencoded; 内容类型:application / x-www-form-urlencoded; charset=UTF-8
字符集= UTF-8
Content-Length: 88 内容长度:88
connect_status=-1; connect_status = -1; __utmb=50703532.0.10.1303366930
__utmb = 50703532.0.10.1303366930
DNT: 1 DNT:1
Connection: keep-alive 连接:保持活动状态
Pragma: no-cache 语法:无缓存
Cache-Control: no-cache 缓存控制:无缓存
method=tagged.usermgmt.addFriend&uid_to_add=5402501977&api_signature=&track=1mJ0lY7-W3 方法= tagged.usermgmt.addFriend&uid_to_add = 5402501977&api_signature =&跟踪= 1mJ0lY7-W3
I understand everything up until the method. 我知道直到方法为止的一切。 As far as I know you can only supply GET or POST to HttpWebRequest.Method.
据我所知,您只能向HttpWebRequest.Method提供GET或POST。 Could someone maybe fill in the blanks for me?
有人可以帮我填补空白吗?
Is it possible to retrieve data from a page that loads content dynamically through javascript by using WebClient/HttpWebRequest?
是否可以从使用WebClient / HttpWebRequest通过javascript动态加载内容的页面中检索数据?
Nope - you can retrieve all of the JavaScript files and other documents required to execute the JavaScript, but the HttpWebRequest
class won't execute that JavaScript for you - this is an altogether more complex task. 不,您可以检索执行 JavaScript所需的所有JavaScript文件和其他文档,但是
HttpWebRequest
类不会为您执行该JavaScript-这是一个更加复杂的任务。
If you wanted to do this then you could host a web browser inside your application, get the browser to open and "render" the page, then inspect the result. 如果要执行此操作,则可以在应用程序中托管Web浏览器,使浏览器打开并“渲染”页面,然后检查结果。 In general however is far easier just to come up with an alternative solution based on the situation, for example if the JavaScript populates the page from an AJAX request then just run the request yourself directly.
但是,总的来说,根据情况提出一种替代解决方案要容易得多,例如,如果JavaScript从AJAX请求中填充页面,则直接自己运行该请求。
Would I be able to use WebClient/HttpWebRequest to replicate a XMLHttpRequest like you would see executed via javascript?
我将能够使用WebClient / HttpWebRequest来复制XMLHttpRequest,就像您将通过javascript执行的那样吗?
If you mean "Can I use HttpWebRequest
to execute a HTTP request similar to an AJAX request executed by client JavaScript" then the answer is yes, however exactly how you do this will depend on the JavaScript and the parameters used to exeucte the AJAX request. 如果您的意思是“我可以使用
HttpWebRequest
来执行与客户端JavaScript执行的AJAX请求类似的HTTP请求”,那么答案是肯定的,但是确切的执行方式将取决于JavaScript和用于执行AJAX请求的参数。
In this situation a web debugging tool (such as Fiddler ) is useful as it allows you to inspect and compare the request being made. 在这种情况下,Web调试工具(例如Fiddler )很有用,因为它允许您检查和比较正在发出的请求。
If you are trying to talk to the server in the same manner that javascript does however you might have some options. 如果您尝试以与javascript相同的方式与服务器通信,则可能会有一些选择。
First, you have to determine how those pages communicate with the server. 首先,您必须确定这些页面如何与服务器通信。 For example, JSON or SOAP.
例如,JSON或SOAP。 You can do this with a network sniffer such as wireshark .
您可以使用诸如Wireshark的网络嗅探器执行此操作。
Once you have done this, you can send a JSON request of your own to that server using and parse the JSON response using one of the available C# JSON parsers such as JSON.NET . 完成此操作后,您可以使用自己的JSON请求发送到该服务器,并使用可用的C#JSON解析器之一(例如JSON.NET)解析JSON响应。
You can use a similar method in the case of SOAP. 对于SOAP,可以使用类似的方法。 Be mindful they might not like you talking to their API's this way.
请注意,他们可能不喜欢您这样谈论他们的API。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.