I am trying to write a little bot that is supposed to access this site http://lsa.colorado.edu/cgi-bin/LSA-pairwise.html , enter some text in the textarea and fetch the resulting page from submitting by pressing the submit button. This is for a linguistics project. However, when I execute the click on the HtmlSubmitInput Button the returned URL seems to be malformed as IncorrectnessListenerImpl notifies me:
Apr 10, 2016 2:38:35 PM com.gargoylesoftware.htmlunit.IncorrectnessListenerImpl notify
WARNUNG: Incorrect URL "http:/cgi-bin/LSA-pairwise-x.html" has been corrected
The URL should be
This then leads to the following stacktrace (shortened due to length):
Exception in thread "main" java.lang.RuntimeException: java.net.UnknownHostException: cgi-bin: unknown error
at com.gargoylesoftware.htmlunit.WebClient.download(WebClient.java:2078)
at com.gargoylesoftware.htmlunit.html.HtmlForm.submit(HtmlForm.java:141)
at com.gargoylesoftware.htmlunit.html.HtmlSubmitInput.doClickStateUpdate(HtmlSubmitInput.java:90)
at com.gargoylesoftware.htmlunit.html.DomElement.click(DomElement.java:795)
at com.gargoylesoftware.htmlunit.html.DomElement.click(DomElement.java:742)
at com.gargoylesoftware.htmlunit.html.DomElement.click(DomElement.java:689)
at LSABot.submitInput(LSABot.java:30)
at Start.main(Start.java:8)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
[...]
My guess is that HtmlUnit tries to fix the URL but this results in only "cgi-bin", which of course is malformed. I've searched over and over but have not found anything relevant for my issue.
My LSABot class:
public class LSABot {
final WebClient webClient;
private HtmlPage mainPg, rsltPg;
private HtmlForm htmlForm;
private HtmlTextArea txtA;
private HtmlSubmitInput submitBt;
public LSABot () throws Exception {
this.webClient = new WebClient(BrowserVersion.CHROME);
this.webClient.getOptions().setJavaScriptEnabled(true);
this.mainPg = this.webClient.getPage("http://lsa.colorado.edu/cgi-bin/LSA-pairwise.html");
this.htmlForm = this.mainPg.getForms().get(0);
this.txtA = this.htmlForm.getTextAreaByName("txt1");
this.submitBt = this.htmlForm.getInputByValue("Submit Texts");
}
public void submitInput(String input) {
this.txtA.setText(input);
try {
this.rsltPg = this.submitBt.click();
this.webClient.waitForBackgroundJavaScript(30*1000);
} catch (IOException ioe) {
ioe.printStackTrace();
}
}
The error comes from the html content of the form. The action
attribute should be http://lsa.colorado.edu/cgi-bin/LSA-pairwise-x.html
instead of http:/cgi-bin/LSA-pairwise-x.html
.
Try this code, it should work:
LogFactory.getFactory().setAttribute("org.apache.commons.logging.Log", "org.apache.commons.logging.impl.NoOpLog");
java.util.logging.Logger.getLogger("com.gargoylesoftware").setLevel(Level.OFF);
java.util.logging.Logger.getLogger("org.apache.commons.httpclient").setLevel(Level.OFF);
WebClient client = new WebClient(BrowserVersion.CHROME);
client.getOptions().setJavaScriptEnabled(true);
client.getOptions().setThrowExceptionOnScriptError(false);
client.getOptions().setThrowExceptionOnFailingStatusCode(false);
String url = "http://lsa.colorado.edu/cgi-bin/LSA-pairwise.html";
final HtmlPage page = client.getPage(url);
HtmlForm htmlForm = page.getForms().get(0);
HtmlTextArea txtA = htmlForm.getTextAreaByName("txt1");
txtA.setText("hello");
HtmlSubmitInput submitBt = htmlForm.getInputByValue("Submit Texts");
// change the form action attribute to the correct one
htmlForm.setAttribute("action", "http://lsa.colorado.edu/cgi-bin/LSA-pairwise-x.html");
HtmlPage page2 = submitBt.click();
System.out.println(page2.asText());
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.