search embeded webpage source in vb.net

Question

I wrote a program that includes an embedded web browser that loads a website which have a changing part (the part changes about 2 times a week and it have no regular timing pattern) that I want to search for a particular part in the opened webpage source code after refreshing the webpage in a specified time interval.

I found many things similar to my question but this is what I want and those questions doesn't have:

• search embedded webpage source (they searching the webpage without embedding, and I had to embed it because I had to login before I see the particular page)

so this is the procedure I'm trying to do:

1- open a website in embedded web browser

2- after user logged in, with a press of button in program, it hides the embedded web browser and start to refresh the page in a time interval (like every minute) and search if the particular code changed in the source of that opened webpage

any other/better Ideas appreciated

thanks

Answer 1

Many years ago I wrote an app to reintegrate forum posts from several pages into one and I struggled with the login issue too and thought it was only possible using an embedded browser. As it turns out, it's possible to use System.Net in .NET to handle web pages that need a login as you can pull the cookies out and keep them on hand. I would suggest you do that and move away from the embedded browser.

Unfortunately I wrote the code in C# originally, but as it's .NET and is mostly classes-based, it shouldn't be too difficult to port over.

The Basic Principle

Find out what information is included in the POST when you login, which you can do in Chrome with developer mode on (F12). Convert that to a byteArray, POST it to the page, store the cookies and make another call with the cookie data later on. You will need a class variable to hold the cookies.

Code:

private void Login()
    {
        byte[] byteArray = Encoding.UTF8.GetBytes("username=" + username + "&password=" + password + "&autologin=on&login=Log+in"); // Found by investigation

        HttpWebRequest request = (HttpWebRequest)WebRequest.Create("yourURL");
        request.AllowAutoRedirect = false;
        request.CookieContainer = new CookieContainer();
        request.Method = "POST";
        request.ContentLength = byteArray.Length;
        request.ContentType = "application/x-www-form-urlencoded";
        Stream dataStream = request.GetRequestStream();
        dataStream.Write(byteArray, 0, byteArray.Length);
        dataStream.Close();
        WebResponse response = request.GetResponse();
        if (((HttpWebResponse)response).StatusCode == HttpStatusCode.Found)
        {
            // Well done, your login has been accepted
            loginDone = true;
            cookies = request.CookieContainer;
        }
        else
        {
            // If at first you don't succeed...
        }

        response.Close();
    }

private string GetResponseHTML(string url)
    {
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
        request.AllowAutoRedirect = false;

        // Add cookies from Login()
        request.CookieContainer = cookies; 

        request.ContentType = "application/x-www-form-urlencoded";
        WebResponse response = request.GetResponse();
        string sResponse = "";
        StreamReader reader = null;
        if (((HttpWebResponse)response).StatusCode == HttpStatusCode.OK)
        {
            reader = new StreamReader(response.GetResponseStream());
            sResponse = reader.ReadToEnd();
            reader.Close();
        }
        response.Close();

        return sResponse;
    }

Hope that helps.

Answer 2

I had to change to C# and I found what I was looking for:

string webPageSource = webBrowser1.DocumentText;

That gave me the source of web page opened in webBrowser1 control.