Auto-complete a HTML form using a C# program

Question

I want to write a C# program (for purely educational purposes) that would auto-complete a HTML form which resides at a specified URL. The thing is I don't know if that's possible or how can I get started? I thought of downloading the HTML, parsing it for the form's elements' names and then make a GET on that URL with the necessary parameters. Is that a good idea?

Answer 1

See System.Net.WebRequest . You can use this to issue GET requests, where variables are encoded in the URL, or POST requests, where the variables go in the body of the request. The HTML source code of the form will tell you what type of request to issue.

You won't need to download or parse the form itself at runtime if the structure of the form doesn't change: you can hard-code the variable names in your program at the point where you construct the GET or POST request.

Answer 2

The general idea that you have is correct, though normally a HTML form will perform a POST back to the server rather than a GET. See this question for example code for a web request in C# that supplies arguments. Pay special attention to the comment telling you to add request.Method = "POST"; since that will change the request from a GET. (You can tell is a form is a GET or POST by looking at the method in it's opening tag. This:

<form id="form" action="..." method="post">

is a POST form. No declared method means that it is a GET)

The more complex problem is parsing the HTML to find the elements. This is a tricky issue since the looseness of the HTML standard means that you can't simply treat it as an XML document and parse it that way, instead you have to come up with a more forgiving method. There are a few questions on how to do this on this site ( this for example ) and it is a common problem, so Google will no doubt give you some leads.

If I was approching this problem I'd get the web request working with a known form first (so you can skip the parsing) and worry about that part once you are comfortable with sending and retrieving data from the server.

Answer 3

WatiN will do this for you, it automates IE for testing purposes. But I have used it to screen scrape things. It works well.

http://watin.sourceforge.net/

[Test] 
public void SearchForWatiNOnGoogle()
{
 using (IE ie = new IE("http://www.google.com"))
 {
  ie.TextField(Find.ByName("q")).TypeText("WatiN");
  ie.Button(Find.ByName("btnG")).Click();

  Assert.IsTrue(ie.ContainsText("WatiN"));
 }
}

Answer 4

Dave Ward wrote a great post that centers around posting back to a page method using AJAX. The idea is to serialize the response objects to JSON and display on the client using jQuery. This post is well written with a great example that can applied to your situation.

If you read through the post you'll see that your idea is aligned with his strategy. Dave's site has many cool articles that blend Asp.net, AJAX, and client techniques using jQuery and Javascript.

Answer 5

Sounds like a screen/web scrape with form submission involved. You may want to check my answer to this question .