Jsoup无法连接（有时）

Question

I'm using jsoup in a project, but I find that sometimes, the following instruction is unable to obtain the document:

 Document document = Jsoup.connect(url).timeout(30000).get();

The strange thing is that with the browser I can open the url in less then 2 seconds, while jsoup has this problem. Another strange thing is that most of the time, jsoup works well.

What's wrong?

Answer 1

Some websites look at the user-agent string of the connecting browser to decide what content to deliver. It may be that the user-agent Jsoup sends along is not clear enough. So my suggestions would be to play with the user-agent like this:

Document document = Jsoup.connect(url)
   .userAgent("Mozilla/5.0 (Windows; U; WindowsNT 5.1; en-US; rv1.8.1.6) Gecko/20070725 Firefox/2.0.0.6")
   .timeout(30000)
   .get();

Another possibility would be that the webserver needs some cookies to be set correctly. You need to look at the exact traffic between a browser and the website to find out more. (Use the Network tab in the browser development mode)

Without the URL that gives you the problems I fear this is all advice I can offer.