JSOUP在Eclipse中抛出url状态503但是URL在浏览器中工作正常

Question

In particular, this is with the website amazon.com to be specific. I am receiving a 503 error for their domain, but I can successfully parse other domains.

I am using the line

Document doc = Jsoup.connect(url).timeout(30000).get();

to connect to the URL.

Answer 1

You have to set a User Agent :

Document doc = Jsoup.connect(url).timeout(30000).userAgent("Mozilla/17.0").get();

(Or others; best you choose a browser user agent)

Else you'll get blocked.

Please see also: Jsoup: select(div[class=rslt prod]) returns null when it shouldn't

Answer 2

you can try

val ret=Jsoup.connect(url)
  .userAgent("Mozilla/5.0 Chrome/26.0.1410.64 Safari/537.31")
  .timeout(2*1000)
  .followRedirects(true)
  .maxBodySize(1024*1024*3)    //3Mb Max
  //.ignoreContentType(true) //for download xml, json, etc
  .get()

it maybe works, maybe amazon.com need followRedirects set to true.