在Java中运行Javascript表单请求

Question

I want to be able to get POST form request information that would return the name of corporations that begin with a certain text field. The site I want to do this from is below:

http://corp.sec.state.ma.us/corpweb/CorpSearch/CorpSearch.aspx

I want to get results for the "Begins with" query. Instead getting back results I get back junk and something that says "Please enable JavaScript to view the page content." from my results. From what I see believe there is possibly one of two problems.

1) I need to enable javascript(if possible) through java

2) I didn't send the request properly and need to change my submit variables

If number 1 is the problem what is the best approach for me to take to accomplish my goal?(another language like php, etc). I have been unable to find an easy solution to this problem.

If number 2 is the problem what variables do I need to change?

The following java code and html below:

HttpClient client = new DefaultHttpClient();
        HttpPost httpPost = new HttpPost(
                "http://corp.sec.state.ma.us/corpweb/CorpSearch/CorpSearch.aspx");
        httpPost.setHeader("Connection", "keep-alive");
        httpPost.setHeader("User-Agent",
                "Mozilla/5.0 (Windows NT 6.1; rv:9.0.1) Gecko/20100101 Firefox/9.0.1");
        httpPost.setHeader("Accept",
                " text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
        httpPost.setHeader("Accept-Language", "en-us,en;q=0.5");
        httpPost.setHeader("Host", "mysite.com");

        ArrayList<NameValuePair> nameValuePair = new ArrayList<NameValuePair>();
        nameValuePair.add(new BasicNameValuePair("ctl00$MainContent$btnSearch" , ""));
        nameValuePair.add(new BasicNameValuePair("ctl00$MainContent$ddBeginsWithEntityName" , "B"));
        nameValuePair.add(new BasicNameValuePair("ctl00$MainContent$CorpSearch" , "rdoByEntityName"));
        nameValuePair.add(new BasicNameValuePair(
                "ctl00$MainContent$txtEntityName", "google"));
        httpPost.setEntity(new UrlEncodedFormEntity(nameValuePair));
        httpPost.setEntity(new UrlEncodedFormEntity(nameValuePair));
        HttpResponse response = client.execute(httpPost);

The Form code I am trying to modify is below:

Search By:

<input id="MainContent_rdoByEntityName" type="radio" name="ctl00$MainContent$CorpSearch" value="rdoByEntityName" checked="checked">

Search Text:

  <input name="ctl00$MainContent$txtEntityName" type="text" value="intelligent" maxlength="175" id="MainContent_txtEntityName" class="p3" onkeyup="javascript: textChangedEntityName(this);" onmousedown="javascript: textChangedEntityName(this);" style="width:90%;">

Search Type:

 <select name="ctl00$MainContent$ddBeginsWithEntityName" id="MainContent_ddBeginsWithEntityName" class="p4" style="width:88%;">
        <option selected="selected" value="B">Begins with</option>
        <option value="M">Exact match</option>
        <option value="F">Full text</option>
        <option value="S">Soundex</option>

    </select>

Search Button:

<input type="submit" name="ctl00$MainContent$btnSearch" value="Search Corporations" onclick="javascript:return loadMaxHeightWidth();WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(&quot;ctl00$MainContent$btnSearch&quot;, &quot;&quot;, true, &quot;&quot;, &quot;&quot;, false, false))" id="MainContent_btnSearch" class="Button" style="width:200px;">

Thank you!!!

Answer 1

Please note that HttpClient is best for raw HTTP communication and not for GUI-less browser with javascript support. You can use selenium which has very good support for javascript or htmlunit

Answer 2

Try out the HTMLUnit library. It can execute javascript and perform actions as if a user is clicking or filling out forms. Its easy to use and to learn.