简体   繁体   English

使用HtppWebRequest发布表单数据没有效果

[英]Posted Form data using HtppWebRequest has not effect

I came across a website which seems simple enough that I was pretty confident that I will be able to read its data using HttpWebRequest and will be able to do the GET and POST requests. 我遇到了一个看起来很简单的网站,我非常有信心能够使用HttpWebRequest读取数据并能够执行GET和POST请求。 The GET requests are working fine. GET请求工作正常。 POST request also not generating any error but still the posted form data has no effect on the results which are returned. POST请求也不会产生任何错误,但仍然发布的表单数据对返回的结果没有影响。 The form data posted have fields to filter the data as per dates but regardless the fact that every required data is posted the data returned is not filtered. 发布的表单数据具有按日期过滤数据的字段,但无论是否发布了所有必需数据,都不会过滤返回的数据。 I have added every header, form data and also added cookies with the request. 我添加了每个标题,表单数据以及添加了请求的cookie。

The url for the webpage is http://www.bseindia.com/corporates/Insider_Trading_new.aspx?expandable=0 该网页的网址是http://www.bseindia.com/corporates/Insider_Trading_new.aspx?expandable=0

This seems like a very ordinary website but as it is an aspx page and involves ViewState and Event Validation hence this was expected not to be very easy. 这看起来像一个非常普通的网站,但由于它是一个aspx页面,涉及ViewState和事件验证,因此预计这不会很容易。

My first step was to analyze the site's GET and POST using Fiddler and this amazes me because Fiddler is not capturing any traffic for this url. 我的第一步是使用Fiddler分析网站的GET和POST,这让我感到惊讶,因为Fiddler没有为这个网址捕获任何流量。 I have tried Charles but that itself is not capturing this url. 我试过查尔斯,但这本身并没有抓住这个网址。 Other then this Url Fiddler and Charles both are capturing everything else. 除此之外,这个Url Fiddler和Charles都捕获了其他一切。 I also like to mention that when I called the Url from a console application using HttpWebRequest then both Fiddler and Charles captured it but they are not capturing it from Chrome, FireFox and Internet Explorer 11. 我还想提一下,当我使用HttpWebRequest从控制台应用程序调用Url时,Fiddler和Charles都捕获了它,但他们没有从Chrome,FireFox和Internet Explorer 11捕获它。

So I analyzed the Network activity using Developer tool in FireFox and everything was visible which includes (Headers, Parameters and Cookies). 因此,我使用FireFox中的开发人员工具分析了网络活动,所有内容都可见,其中包括(标题,参数和Cookie)。 In Chrome no cookies were present. 在Chrome中,没有Cookie存在。 When I inspect the cookies by creating HttpWebRequest and got the response there were no cookies present. 当我通过创建HttpWebRequest检查cookie并得到响应时,没有cookie存在。 So something is really strange going o with this website. 所以这个网站真的很奇怪。

I have somehow managed to create a simple function to create the request and get the response. 我以某种方式设法创建一个简单的函数来创建请求并获得响应。 What I am doing is that I am creating a GET request first and get the Website string and extract Viewstate, EventValidation etc from it. 我正在做的是我首先创建一个GET请求并获取网站字符串并从中提取Viewstate,EventValidation等。 I use this information to be used in second HttpWebRequest which is a post. 我使用此信息在第二个HttpWebRequest中使用,这是一个帖子。 Now everything works fine and I get the response but not as expected. 现在一切正常,我得到的回应却不如预期。 I want the records between two give dates and I have specified these dates in the form data but still the POST request does not return the filtered data. 我想要两个给出日期之间的记录,我已经在表单数据中指定了这些日期,但POST请求仍然不返回过滤后的数据。 I have mentioned the function that I have created below and I will really appreciate any suggestions that why is this happening and how to handle this. 我已经提到了我在下面创建的函数,我将非常感谢为什么会发生这种情况以及如何处理它的任何建议。 To understand this has become a challenge to me as I cannot understand why this simple website is not showing up in Fiddler. 要理解这一点对我来说是一个挑战,因为我无法理解为什么这个简单的网站没有出现在Fiddler中。 (This uses Javascript Postback) (这使用Javascript Postback)

The code may look long and scary but rather it is very simple and straight forward. 代码可能看起来很长而且很可怕,但它非常简单直接。

Try

        ' First GET Request to obtain Viewstate, Eventvalidation etc
        Dim objRequest2 As Net.HttpWebRequest = DirectCast(HttpWebRequest.Create("http://www.bseindia.com/corporates/Insider_Trading_new.aspx?expandable=0"), HttpWebRequest)
        objRequest2.Method = "GET"
        objRequest2.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
        objRequest2.Headers.Add("Accept-Encoding", "gzip, deflate")
        objRequest2.Headers.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6,ur;q=0.4")
        objRequest2.KeepAlive = True
        objRequest2.ContentType = "application/x-www-form-urlencoded"
        objRequest2.Host = "www.bseindia.com"
        objRequest2.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"

        objRequest2.AutomaticDecompression = DecompressionMethods.Deflate Or DecompressionMethods.GZip

        Dim LoginRes2 As Net.HttpWebResponse
        Dim sr2 As IO.StreamReader
        LoginRes2 = objRequest2.GetResponse()

        sr2 = New IO.StreamReader(LoginRes2.GetResponseStream)
        Dim getString As String = sr2.ReadToEnd()
        Dim getCookieCollection = objRequest2.CookieContainer

        ' get the page ViewState                
        Dim viewStateFlag As String = "id=""__VIEWSTATE"" value="""
        Dim i As Integer = getString.IndexOf(viewStateFlag) + viewStateFlag.Length
        Dim j As Integer = getString.IndexOf("""", i)
        Dim viewState As String = getString.Substring(i, j - i)

        ' get page EventValidation                
        Dim eventValidationFlag As String = "id=""__EVENTVALIDATION"" value="""
        i = getString.IndexOf(eventValidationFlag) + eventValidationFlag.Length
        j = getString.IndexOf("""", i)
        Dim eventValidation As String = getString.Substring(i, j - i)

        ' get page EventValidation                
        Dim viewstateGeneratorFlag As String = "id=""__VIEWSTATEGENERATOR"" value="""
        i = getString.IndexOf(viewstateGeneratorFlag) + viewstateGeneratorFlag.Length
        j = getString.IndexOf("""", i)
        Dim viewStateGenerator As String = getString.Substring(i, j - i)

        viewState = System.Web.HttpUtility.UrlEncode(viewState)
        eventValidation = System.Web.HttpUtility.UrlEncode(eventValidation)

        Dim LoginRes As Net.HttpWebResponse
        Dim sr As IO.StreamReader
        Dim objRequest As Net.HttpWebRequest

        ' Second POST request to post the form data along with cookies
        objRequest = DirectCast(HttpWebRequest.Create("http://www.bseindia.com/corporates/Insider_Trading_new.aspx?expandable=0"), HttpWebRequest)

        Dim formDataCollection As New NameValueCollection

        formDataCollection.Add("__EVENTTARGET", "")
        formDataCollection.Add("__EVENTARGUMENT", "")
        formDataCollection.Add("__VIEWSTATE", viewState)
        formDataCollection.Add("__VIEWSTATEGENERATOR", viewStateGenerator)
        formDataCollection.Add("__EVENTVALIDATION", eventValidation)
        formDataCollection.Add("fmdate", "20160104")
        formDataCollection.Add("eddate", "20160204")
        formDataCollection.Add("hidCurrentDate", "2016/02/04")
        formDataCollection.Add("ctl00_ContentPlaceHolder1_hdnCode", "")
        formDataCollection.Add("txtDate", "04/01/2016")
        formDataCollection.Add("ddlCalMonthDiv3", "1")
        formDataCollection.Add("ddlCalYearDiv3", "2016")
        formDataCollection.Add("txtTodate", "04/02/2016")
        formDataCollection.Add("ddlCalMonthDiv4", "2")
        formDataCollection.Add("ddlCalYearDiv4", "2016")
        formDataCollection.Add("Hidden1", "")
        formDataCollection.Add("ctl00_ContentPlaceHolder1_GetQuote1_smartSearch", "Enter Security Name / Code / ID")
        formDataCollection.Add("btnSubmit.x", "44")
        formDataCollection.Add("btnSubmit.y", "2")

        Dim strFormdata As String = formDataCollection.ToString()
        Dim encoding As New ASCIIEncoding
        Dim postBytes As Byte() = encoding.GetBytes(strFormdata)

        objRequest.Method = "POST"
        objRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
        objRequest.Headers.Add("Accept-Encoding", "gzip, deflate")
        objRequest.Headers.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6,ur;q=0.4")
        objRequest.Headers.Add("Cache-Control", "private, max-age=60")
        objRequest.KeepAlive = True
        objRequest.ContentType = "application/x-www-form-urlencoded"
        objRequest.Host = "www.bseindia.com"
        objRequest.Headers.Add("Origin", "http://www.bseindia.com")
        objRequest.Referer = "http://www.bseindia.com/corporates/Insider_Trading_new.aspx?expandable=0"
        objRequest.Headers.Add("Upgrade-Insecure-Requests", "1")
        objRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"

        objRequest.ContentType = "text/html; charset=utf-8"
        objRequest.Date = "Thu, 04 Feb 2016 13:42:04 GMT"
        objRequest.Headers.Add("Server", "Microsoft-IIS/8.0")
        objRequest.Headers.Add("Vary", "Accept-Encoding")
        objRequest.Headers.Add("X-AspNet-Version", "2.0.50727")
        objRequest.Headers.Add("ASP.NET", "ASP.NET")

        objRequest.AutomaticDecompression = DecompressionMethods.Deflate Or DecompressionMethods.GZip

        Dim gaCookies As New CookieContainer()

        Dim cookie1 As New Cookie("__asc", "f673f0d5152a823bc335f575d34")
        cookie1.Domain = ".bseindia.com"
        cookie1.Path = "/"
        gaCookies.Add(cookie1)

        Dim cookie2 As New Cookie("__auc", "f673f0d5152a823bc335f575d34")
        cookie2.Domain = ".bseindia.com"
        cookie2.Path = "/"
        gaCookies.Add(cookie2)

        Dim cookie3 As New Cookie("__utma", "253454874.280640365.1454519857.1454519865.1454519865.1")
        cookie3.Domain = ".bseindia.com"
        cookie3.Path = "/"
        gaCookies.Add(cookie3)

        Dim cookie4 As New Cookie("__utmb", "253454874.1.10.1454519865")
        cookie4.Domain = ".bseindia.com"
        cookie4.Path = "/"
        gaCookies.Add(cookie4)

        Dim cookie5 As New Cookie("__utmc", "253454874")
        cookie5.Domain = ".bseindia.com"
        cookie5.Path = "/"
        gaCookies.Add(cookie5)

        Dim cookie6 As New Cookie("__utmt", "1")
        cookie6.Domain = ".bseindia.com"
        cookie6.Path = "/"
        gaCookies.Add(cookie6)

        Dim cookie7 As New Cookie("__utmz", "253454874.1454519865.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)")
        cookie7.Domain = ".bseindia.com"
        cookie7.Path = "/"
        gaCookies.Add(cookie7)

        Dim cookie8 As New Cookie("_ga", "GA1.2.280640365.1454519857")
        cookie8.Domain = ".bseindia.com"
        cookie8.Path = "/"
        gaCookies.Add(cookie8)

        Dim cookie9 As New Cookie("_gat", "1")
        cookie9.Domain = ".bseindia.com"
        cookie9.Path = "/"
        gaCookies.Add(cookie9)

        Dim postStream As Stream = objRequest.GetRequestStream()
        postStream.Write(postBytes, 0, postBytes.Length)
        postStream.Flush()
        postStream.Close()

        LoginRes = objRequest.GetResponse()
        sr = New IO.StreamReader(LoginRes.GetResponseStream)

        ReadWebsite = sr.ReadToEnd()

        sr.Close()
        sr = Nothing
        LoginRes.Close()
        LoginRes = Nothing
        objRequest = Nothing
        Exit Function

    Catch ex As Exception
        ReadWebsite = Nothing
    End Try

Note: (Raw form data for dates without viewstate and eventvalidation) 注意:(没有viewstate和eventvalidation的日期的原始表单数据)

fmdate:20160130 eddate:20160205 hidCurrentDate:2016/02/05 ctl00_ContentPlaceHolder1_hdnCode: txtDate:04/01/2016 ddlCalMonthDiv3:1 ddlCalYearDiv3:2016 txtTodate:04/02/2016 ddlCalMonthDiv4:2 ddlCalYearDiv4:2016 Hidden1: ctl00_ContentPlaceHolder1_GetQuote1_smartSearch:Enter Security Name / Code / ID btnSubmit.x:55 btnSubmit.y:13 fmdate:20160130 eddate:20160205 hidCurrentDate:2016/02/05 ctl00_ContentPlaceHolder1_hdnCode:txtDate:04/01/2016 ddlCalMonthDiv3:1 ddlCalYearDiv3:2016 txtTodate:04/02/2016 ddlCalMonthDiv4:2 ddlCalYearDiv4:2016 Hidden1:ctl00_ContentPlaceHolder1_GetQuote1_smartSearch:输入安全名称/代码/ ID btnSubmit.x:55 btnSubmit.y:13

You could consider running the site in a browser and using a tool to control the browser instead directly issuing GET/POST requests. 您可以考虑在浏览器中运行该站点并使用工具来控制浏览器,而不是直接发出GET / POST请求。 This may be easier and slightly more robust than your current approach. 这可能比您当前的方法更容易,也更稳健。

Eg Selenium Web Driver http://www.seleniumhq.org/projects/webdriver/ 例如Selenium Web Driver http://www.seleniumhq.org/projects/webdriver/

You would load the page, set the values of the form fields (using css style selectors to find the appropriate fields) and then click the button. 您将加载页面,设置表单字段的值(使用css样式选择器查找相应的字段),然后单击按钮。 You can automate all of this and get the page source (unfortunately I don't think you can get the full html in it's current state, after javascript has run, but potentially you can use the api to get the elements you need). 你可以自动完成所有这些并获取页面源代码(不幸的是,我认为你不能在javascript运行后获得当前状态的完整html,但可能你可以使用api来获取你需要的元素)。

Api documentation: http://seleniumhq.github.io/selenium/docs/api/dotnet/ Api文档: http//seleniumhq.github.io/selenium/docs/api/dotnet/

You indeed should include ALL fields from the form, including hidden ones and ASP session identifier that is stored in cookies. 确实应该包括表单中的所有字段,包括隐藏的字段和存储在cookie中的ASP会话标识符。 That way you fully emulate browser' request and achieve your goal. 这样您就可以完全模仿浏览器的请求并实现目标。 To show what you have to submit - http://pastebin.com/AsSABgU6 要显示您要提交的内容 - http://pastebin.com/AsSABgU6

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM