使用Jsoup从页面获取元素

Question

I want to log in to a https website using Jsoup and make subsequent calls 3-4 services to check whether a job is done or not.

    public class JSOUPTester {
    public static void main(String[] args){
        System.out.println("Inside the JSOUP testing method");
        String url = "https://someloginpage.com";
    try{
        Document doc = Jsoup.connect(url).get();
        String S = doc.getElementById("username").text();// LINE 1
        String S1 = doc.getElementById("password").text();// LINE 2
    }catch(Exception e){
         e.printStackTrace();
     }
   }
}

Exception:

java.lang.NullPointerException
JSOUPTester.main(JSOUPTester.java:7)

I have checked in the chrome that these pages contain elements with id "username" and "password". The lines above are throwing NullPointerException. What I am doing wrong here?

Answer 1

A Number of things can be the cause of this. Without the URL I can't be certain, but here are some clues:

• Some pages load their content via AJAX. Jsoup can#t deal with this, since it does not interpret any JavaScript. You can check for this by downloading the page with curl, or in a browser while turnig off JavaScript. To deal with pages that use JavaScript to render themselves, you can use tools like Selenium webdriver or HTMLUnit.

• The webserver of the page that you try to load might require a cookie to be present. You need to look at the network traffic that happens surfing loading of that page. In chrome or firefox you can see this in the developer tools in the network tab.

• The webserver might respond differently for different clients. That is why you may have to set the UserAgent string to a known Browser in your JSoup http request.

  Jsoup.connect("url").userAgent("Mozilla/5.0")

• JSoup has a size limitation of 1MB for the downloaded html source. You can turn this off or set it to a larger value if needed.

  Jsoup.connect("url").maxBodySize(0)

• Jsoup might timeout on the request. To change timeout behavior use

  Jsoup.connect("url").timeout(milliseconds)

• There might be other reasons I did not think of now.