vb.NET WebRequest to read aspx page to string, access denied?

Question

I'm trying to make an executable in VS2008 that will read a webpage source code using a vb.NET function into a string variable. The problem is that the page is not *.html but rather *.aspx.

I need a way to execute the aspx and get the displayed html into a string.

The page I want to read is any page of this type: http://www.realtor.ca/PropertyDetails.aspx?PropertyID=9620716

I have tried the following code, which works properly for html pages, but generates the wrong source code with "access denied" for the page title when I pass in the above aspx page.

    Dim myReq As WebRequest = WebRequest.Create(url)

    Dim myWebResponse As WebResponse = myReq.GetResponse()

    Dim dataStream As Stream = myWebResponse.GetResponseStream()

    Dim reader As New StreamReader(dataStream, System.Text.Encoding.UTF8)

    Dim responseFromServer As String = reader.ReadToEnd()

Any suggestions or ideas?

Answer 1

I get the same thing while running wget from the command line:

wget http://www.realtor.ca/PropertyDetails.aspx?PropertyID=9620716

I guess the server is relying on that something is set in the browser before the response is delivered, eg a cookie. You might want to try using a WebBrowser control (you don't have to have it visible) in the following way (this works):

Public Class Form1

    Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
        AddHandler WebBrowser1.DocumentCompleted, New WebBrowserDocumentCompletedEventHandler(AddressOf DocumentCompletedHandler)
        WebBrowser1.Navigate("http://www.realtor.ca/PropertyDetails.aspx?PropertyID=9620716")
    End Sub

    Private Sub DocumentCompletedHandler(ByVal sender As Object, ByVal e As WebBrowserDocumentCompletedEventArgs)
        Console.WriteLine(WebBrowser1.DocumentText)
    End Sub
End Class