C＃自动执行表单登录和数据挖掘

Question

I have a list of username and password in my database as provided by my users, I wish to use these records to automate login to a 3rd party website, and then go to certain page and selection and data mining the HTML response of that particular page. In short, the process as below

Go To Login Page -> Fill in the username and password, select dropdown option and Submit -> Select certain dropdown -> Submit the selection -> Data mining the HTML response -> Repeat the process with the next username and password.

[UPDATE] I have learnt how to do web scraping in .Net, which solve the data mining part.

What I still missing is the capability to automate login (fill in username, password and submit). If server is keep session id as the login info, thus I will need to be able to user the same session id to login and do scrapping.

Answer 1

I wrote an application that would send you an email or text message if the value on a small section of a web page changed. An example would be when your competitors price box for a given product changed - some situations were behind a login page.

As seen below, I entered the URL of the login page on the first row and clicked the "Get" button for that row. It went out and got the page and then populated the "Inputs" grid below with all of the controls on that page.

I then have to enter the values that will be posted back. A radio button needs to be selected for which button is tagged as the button doing the post back. So your process has to be posting back the same input names as the page has and the correct button name.

For this to work, cookies have to be enabled. The login step will return the cookies with a security token added, the second step has to send back cookies with the same token. This is how it knows you are logged in. The tokens have an expire time on them.

To see what this looks like, do it once while watching the traffic with Fiddler.

I wrote this in C#. When a simple site I used the HttpWebRequest object. When Ajax was involved I used the C# WebBrowser object so I could wait for the full page to load, a complication you might have to deal with.

Answer 2

Got the solution, please refer to my code below, this is sufficient to expand into more complex code.

private void bttExecute_Click(object sender, EventArgs e)
        {
            webBrowser1.Navigate(URL);            
        }

        private void webBrowser1_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
        {
            step++;

            switch (step)
            {
                case 1: //Login step                    

                    webBrowser1.Document.GetElementById("txtUsername").InnerText = "user123";

                    //This piece of code auto select the radio selection
                    HtmlElementCollection theElementCollection = webBrowser1.Document.GetElementsByTagName("input");
                    foreach (HtmlElement curElement in theElementCollection)
                    {
                        if (curElement.Id == "radio123")
                        {
                            curElement.InvokeMember("click");

                            break;
                        }
                    }

                    webBrowser1.Document.GetElementById("dropdown").SetAttribute("value", "2015"); 
                    webBrowser1.Document.GetElementById("button").InvokeMember("click");

                    break;

                case 2: 

                    //Grab the source code and process further
                    string sourceCode = webBrowser1.DocumentText;

                    break;

                case 3:



                    break;
            }

        }

Answer 3

After user_1 logon success, you want call back interface logon for user_2 by: window.location = "Enter Url your page html logon"; Ex php: echo("window.location=' http://10.224.41.131/barcode_scan/main.html ';");