使用jsoup解析亚马逊页面返回204状态

Question

sample page: http://www.amazon.com/gp/offer-listing/1589942140

public void connect( String url ) {        
    this.conn = Jsoup.connect( url );  
}

/**
 * Executes the request and parses the result.
 * @return 
 */
public boolean parse() 
{
    try {
        this.page = this.conn.get();
        return true;
    } catch (IOException ex) {
        // log it here
        System.out.format("Error: %s%n", ex);
        return false;
    }
}    

parsing the page creates ioexception below:

org.jsoup.HttpStatusException: HTTP error fetching URL. Status=204, URL= http://www.amazon.com/gp/offer-listing/1589942140

i tried it with the native java url class below and it's not creating IOException:

    try {
        URL myURL = new URL("http://www.amazon.com/gp/offer-listing/1589942140");
        URLConnection myURLConnection = myURL.openConnection();
        myURLConnection.connect();
        System.out.format("%s", myURLConnection.getContentType());
    } 
    catch (MalformedURLException e) { 
        // new URL() failed
        System.out.format("Error: %s%n", e);
    } 
    catch (IOException e) {   
        // openConnection() failed
        System.out.format("Error: %s%n", e);
    }

any ideas why this is so ?

Answer 1

The following works for me:

            System.out.println(Jsoup.connect("http://www.amazon.com/gp/offer-listing/1589942140").userAgent("Mozilla").get().text());;

The URL tried above was as specified by you above. (sample page: http://www.amazon.com/gp/offer-listing/1589942140 )