尽管检查了状态代码，但jsoup仍抛出204状态

Question

While i connect to a url through jsoup. Here is the snippet of my code:

  for (int j = 0; j < unq_urls.size(); j++) {

      Response response2 = Jsoup.connect(unq_urls.get(j))
             .userAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21")
             .timeout(100*1000)
             .ignoreContentType(true)
             .execute();

      if (response2.statusCode() == 200) {
...}

}

When the connection is executed jsoup throws the following error:

org.jsoup.HttpStatusException: HTTP error fetching URL. Status=204, URL=https://www.google.com/gen_204?reason=EmptyURL
    at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:459)
    at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:475)
    at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:475)
    at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:434)
    at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:181)
    at cseapiandparsing.CSE_Author_Name_Dis.<init>(CSE_Author_Name_Dis.java:187)
    at cseapiandparsing.CSE_Author_Name_Dis.main(CSE_Author_Name_Dis.java:263)

How can I overcome this? I mean i want jsoup to pass another URL if it cannot connect to a specific URL. Related to this jsoup also throws time out error when it takes too much time to connect a URL. To this end I have already put .timeout(100*1000) option. However, I was wondering is there a way of passing to another URL if the attempt for the current one takes too long?

Thanks in advance.

Answer 1

I believe you are looking for a try-catch mechanism here.

Surround the Jsoup.connect part with a try clause, then in your catch clause handle the exceptions gracefully, which in your case would be continuing to the next loop.

To skip the current one if it takes too long, simply set timeout() value to your desired waiting period, if it passes that period it will throw a timeout exception, which again will be caught by the catch clause. Try the code I posted below:

for (int j = 0; j < unq_urls.size(); j++) {
  try{
      Response response2 = Jsoup.connect(unq_urls.get(j))
         .userAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21")
         .timeout(100*1000)
         .ignoreContentType(true)
         .execute();
  } catch(Exception e) {
      continue; //continue to the next loop if exception occurs
  }
}